twitter_ebooks 3.0

Version 3.0 of twitter_ebooks is out! New stuff:

  • About 80% less memory and storage use for models
  • Bots run in their own threads (no eventmachine), and startup is parallelized
  • Bots start with ebooks start, and no longer die on unhandled exceptions
  • ebooks auth command will create new access tokens, for running multiple bots
  • ebooks console starts a ruby interpreter with bots loaded (see Ebooks::Bot.all)
  • Replies are slightly rate-limited to prevent infinite bot convos
  • Non-participating users in a mention chain will be dropped after a few tweets
  • API documentation and tests

This should be the last major version release. I didn't expect to put this much work into a twitterbot library! Since a fair few people use it, though, I figured I owe it to them to make sure it behaves itself.


An idea dump for stuff I'd like to try doing someday. This is periodically updated. Not filtered for plausibility!


  • Write a bot which plays Spelunky and tweets about it

From this conversation. Writing a bot which successfully plays Spelunky at all is hard, since it is a complex game which requires navigating a destructable environment and punishes mistakes very harshly. There is a paper discussing this and providing an API; bots are written in C++ or GML.

Once you have a bot, making it describe its experiences could range from simple to tremendously complex. You could fake it with some pregenerated lines ("I defeated a snake on level [x]!") and have it be perfectly entertaining, but translating a variable sequence of events into a natural language narrative is a deep general problem.

Recent years have seen some cool developments in machine learning, particularly image recognition. Google created a model which was able to discover the notion of image generalities like "cats" and "human faces" without even being told to look for these. I think it would be a fun learning experience to train a classifier using data labeled for an ambiguous concept like cuteness, which is a human-assigned property associated with an image that can be quantified but not really associated with one specific set of image features.

To quantify cuteness, use a side-by-side comparison system where users are presented with two semi-randomly selected images and asked to select which is cuter. I think Randall Munroe did this at one point, there's an algorithm somewhere. By only requiring them to select the cuter image, users make binary choices at any given step but you still end up with a sorted hierarchy of images.

(n.b. I feel I should disclaim that cuteness is a subjective quality and what you would be measuring is "consensus cuteness" or the common denominator of what people find cute, there is no true total ordering of cuteness independent of individual experience)

I suspect you would have lots of overfitting problems with the model-- it'd be very easy to train something that only recognizes kittens and gets stuck at a local optimum because kittens are reliably cute.


  • Procedural generation of underlying game mechanics

I remember a persistent myth about Pokémon games involving Mew and trucks. It's hard to imagine this forming with modern games unless they are truly obscure, because only one person needs to understand an aspect of the mechanics and write about it somewhere searchable for everyone to know.1

Procedural generation tackles this by making everyone's game different, so that the past experience of others is not as solid a guide and you have to explore for yourself. However, even games that make heavy use of procgen only really use it for the surface world, the part that is visible to the player. A dragon in nethack always behaves predictably like a dragon, even if the dungeon in which it appears is configured slightly differently.

I think a game which applied randomization even to invisible aspects of the game model would be especially interesting, because it would encourage players to do their own research and uncover things about the universe for themselves.

Important consideration: the variation has to matter. For example, Starbound does procedural generation of enemies but you don't really pay much attention to it because you're mostly just going to shoot at them either way. You have to reward the player for learning things by allowing their optimal strategy to be dramatically changed by new information.

  • A creature breeding game inspired by real genetics

An idea from years ago when I was studying biology. Ties into the above idea about uncovering hidden game mechanics. The obvious point of comparison is the Creatures series, which did a nice job of implementing basic biochemistry by modelling organs as bags which both produced and contained a certain quantity of chemicals.

How could you improve upon Creatures? Well, it'd be nice if there was more game to it. Creatures was a sandbox simulation in the truest sense, and didn't provide much in the way of goals to pursue. There are a few ways you could solve this; the two that come to mind are a multiplayer competitive element like Pokémon or a dynamic, hostile environment like Dwarf Fortress. Creatures would make very interesting elements in a trading game.

Another example of this kind of game is Sonic Adventure 2, in the form of the Chao. Chao are particularly cool because they have subtle gradients of morphological variation based on a combination of genetics and environment, which solves the other issue I had: Creatures just don't look very interesting. Breeding is much more fun when it produces distinct visual forms.

In practice, you'd probably have simplified chromosomes as linear collections of "genes" which produce "proteins" at a given rate in particular parts of the body. The rates could be subject to mutation, and perhaps the protein properties; there's a lot of room for varying how simple or complex the proteins and their interactions are. Abstract physiology is actually pretty straightforward to implement compared to the messy work of producing a game world and graphics.

  • A deterministic tactical RPG in the spirit of Fire Emblem

Some of my fondest memories are of JRPGs which involve teams of characters moving on a 2D grid, a system which ultimately derives from chess. Unlike chess, these games always include a pseudorandom element, like probability of an attack missing or variation in damage. I find this encourages less depth of thought, because no matter how well-considered your plan is there's always some chance of it just failing at any given step for unfair reasons.

Chess shows that it is possible to make an enjoyable game like this without including random elements during gameplay. I think it would be an interesting challenge to design a more complex game along similar lines.

This is one of the ideas I made a start on: I have lots of other thoughts about fun variations and story settings for TRPGs so might come back to this one, especially since a basic Fire Emblem-like system is fairly easy to implement.

HaxeFlixel is sorta a spiritual successor to Flash for game development. I like C# as a language a bit better than Haxe, but it's still pretty nice and the environment is much friendlier than Monogame. (i.e. actual documentation!)

  • Make a Non-Lethal Combat mod for Starbound

Starbound exploration kinda mostly involves beating up native animals and taking people's stuff, which is weird for a game otherwise about cute fuzzy space animal people building things in space. I think adding more variety to combat aside from damaging attacks would also make it more fun!

Immobilizing weapons

Most obvious kind! There's a great tractor beam weapon on the Rho mech in the XS Corporation Mechs mod, could use similar bubble-y effects to hold a creature in place, send it flying away, etc. A weapon which force-activates the gravity techs on enemies would be very amusing, now that I think about it.

Better defenses

Bubble techs which cause attacks/enemies to bounce off! Maybe placeable obstacle type ones as well. Could perhaps use something like the existing "slow" effect on oil to make time distortion fields, or abilities which rearrange blocks around you automatically to make physical cover.

Stealthy things

Might be tricky to implement proper stealth platformer mechanics, but even just a basic energy-draining cloaking tech that just drops you out of combat while it's active would be great.

How to reward player?

The game incentivizes killing stuff with pixel rewards, hunting drops etc. One way we could encourage pacifism is by making the non-lethal weapons just more effective e.g. a stasis field gun that removes smaller enemies from combat quickly so you can get back to exploring. I found myself using paralysis in Skyrim for this a lot.

If possible, it'd be neat if monster types you're consistently peaceful towards become consistently peaceful towards you as well, and perhaps ally with you against hostile enemies or otherwise behave in obviously friendly ways. And enemies you kill a lot of could get tougher and more numerous and start ganging up on you :3

Web Development

  • Add search functionality to the Ghost admin panel

The way I use Ghost is closer to a semi-public Evernote than a blogging engine, and it would be extremely useful to be able to quickly search through posts from the admin panel. I expect this would be handy for normal blogs with a significant number of entries as well, since you do need to edit stuff in the distant past on occasion.

  • A file syncing service which supports image tagging and source annotation

There's lots of lovely artwork and interesting information in the form of image files on the internet. I like to collect these and share them with people. My current solution is just saving stuff to Dropbox, which is fast because saving a file to a folder from a browser is fast and I do not have to wait for the Dropbox servers to do anything. However, Dropbox is not optimized for the purpose of image collections unlike systems like Zerochan, so it's difficult to sort through them later and provide attribution to the creators.

  1. Melissa rightly points out that this may be more a consequence of us having been small children than how little access to the internet we had. A modern example is Herobrine. Still, it'd be nice to be able to create the same sense of mystery in adults, extelligence and all.

Frivolous things

Reading this article about the limited scope of dress for female academics made me think a bit about what it means for something to be frivolous.

When I was 13, I felt very sure that I knew what frivolous was. It was Final Fantasy X-2, to my teenage mind an irredeemable sacrilege of a sequel to the bestest and most amazingest game of all time. Final Fantasy X was a game about love, dreams and the cultural consequences of giant invincible flying doom monsters. X-2 seemed to involve a lot of pop singing and girls in fancy dresses.

Shortly thereafter I discovered my sexuality and spent a few years semi-closeted trying to be more bi than I was. I had no direct objection to being attracted to guys, or even much of an interest in masculinity. I just really didn't want to be seen as frivolous, and I associated that with being openly gay.

You might notice that what we're calling frivolous is really just cultural femininity. The google dictionary definition will even use it in a sentence for you, denouncing the evils of ribbons and frills:

At some point I realized how extremely selective and specific this is. It just so happens that frills are frivolous and not, say, neckties? No impartial analysis of the distribution of human attention would ever come to the conclusion that the most serious distraction is how much effort they put into wearing ribbons!

The true waste was all that worrying about whether things were frivolous or not and whether I might be perceived as such. If a woman wears a bright blue top instead of a grey one, she has expended nobody's time and energy in doing so. When a potential employer lowers their estimations of her professional ability as a result, and fails to hire her into a position for which she is ideal? A huge amount of everyone's time and energy has been wasted! It's absurdly inefficient!

In this sense, people who engage in unusual and colorful self-expression are doing something extremely socially important. They're challenging norms and standards of behavior which are distractingly frivolous. Perhaps once they are dispensed with we can all get on with doing fun and interesting stuff!

Memory profiling in Ruby

The easiest approach is just calling out to ps with the current pid and receiving the resident set size (amount of physical memory allocated to the process).

def Process.rss; `ps -o rss= -p #{}`.chomp.to_i; end  

If you're only interested in a temporary heuristic for debugging a particular issue, this is probably fine. It's platform-specific, though, and you don't have any guarantees about what the garbage collector is doing between calls.

You can use ruby-prof, but measuring memory with it requires patching the Ruby interpreter.

There's also the memory_profiler gem, which uses the ObjectSpace allocation tracing API introduced in 2.1. Since this tracks allocations by origin, it can be resource intensive; in my particular case I found it used more memory than what it was profiling. It's also a young gem and still a bit buggy.

I ended up extracting the core of memory_profiler into a more basic thing which just looks at the total amount of memory allocated over the course of a block, and so is particularly suitable for unit tests:

require 'objspace'                                                              

module MemoryUsage  
  MemoryReport =                                     

  def self.full_gc                                                              
    GC.start(full_mark: true)                                                   

    rvalue_size = GC::INTERNAL_CONSTANTS[:RVALUE_SIZE]                          


    total_memsize = 0                                                           

    generation = nil                                                            
    ObjectSpace.trace_object_allocations do                                     
      generation = GC.count                                                                                                                  

    ObjectSpace.each_object do |obj|                                            
      next unless generation == ObjectSpace.allocation_generation(obj)          
      memsize = ObjectSpace.memsize_of(obj) + rvalue_size                       
      # compensate for API bug                                                  
      memsize = rvalue_size if memsize > 100_000_000_000                        
      total_memsize += memsize                                                  



Extending the Markdown syntax in Ghost

I'm writing a somewhat lengthy thing which really wants footnotes, but Ghost doesn't have any native syntax for them yet. You can put them in manually using html, but it is tedious, and troublesome to reorder. Fortunately it wasn't too hard to add my own hacky1 implementation2:

// Adds footnote syntax as per Markdown Extra:
// That's some text with a footnote.[^1]
// [^1]: And that's the footnote.
//     That's the second paragraph.
// Also supports [^n] if you don't want to worry about preserving
// the footnote order yourself.

(function () {
    var footnotes = function () {
        return [
            { type: 'lang', filter: function(text) {
                var preExtractions = {},
                    hashID = 0;

                function hashId() {
                    return hashID++;

                // Extract pre blocks
                text = text.replace(/```[\s\S]*?\n```/gim, function (x) {
                    var hash = hashId();
                    preExtractions[hash] = x;
                    return "{gfm-js-extract-pre-" + hash + "}";
                }, 'm');

                // Inline footnotes e.g. "foo[^1]"
                var i = 0;
                var inline_regex = /(?!^)\[\^(\d|n)\]/gim;
                text = text.replace(inline_regex, function(match, n) {
                    // We allow both automatic and manual footnote numbering
                    if (n == "n") n = i+1;

                    var s = '<sup id="fnref:'+n+'">' +
                              '<a href="#fn:'+n+'" rel="footnote">'+n+'</a>' +
                    i += 1;
                    return s;

                // Expanded footnotes at the end e.g. "[^1]: cool stuff"
                var end_regex = /\[\^(\d|n)\]: ([\s\S]*?)\n(?!    )/gim;
                var m = text.match(end_regex);
                var total = m ? m.length : 0;
                var i = 0;

                text = text.replace(end_regex, function(match, n, content) {
                    if (n == "n") n = i+1;

                    content = content.replace(/\n    /g, "<br>");

                    var s = '<li class="footnote" id="fn:'+n+'">' +
                              '<p>'+content+'<a href="#fnref:'+n +
                                '" title="return to article"> ↩</a>' +
                              '</p>' +

                    if (i == 0) {
                        s = '<div class="footnotes"><ol>' + s;

                    if (i == total-1) {
                        s = s + '</ol></div>';

                    i += 1;
                    return s;

                // replace extractions
                text = text.replace(/\{gfm-js-extract-pre-([0-9]+)\}/gm, function (x, y) {
                    return preExtractions[y];

                return text;

    // Client-side export
    if (typeof window !== 'undefined' && window.Showdown && window.Showdown.extensions) {
        window.Showdown.extensions.footnotes = footnotes;
    // Server-side export
    if (typeof module !== 'undefined') {
        module.exports = footnotes;
  1. Please don't do this with regexes unless you have to, kids.

  2. Can be found on a branch.

OSW 2014 Update

The open science workshop went well! I gave my talk about SciRate and improving the way people do science. Slides are up here. Also doubled as a useful opportunity to test the site on people in person, and I got some good usability feedback.

Some of the other interesting projects I learned about:

  • SageMathCloud is a computational mathematics tool based on the Python-based Sage framework, and lets you collaboratively edit Sage worksheets and IPython notebooks.

  • Authorea is a web-based paper authoring tool supporting LaTeX and Markdown which uses git as a backend. It takes a lot of inspiration from GitHub, with unlimited free public projects and paid private ones.

  • NBViewer is a simple tool for taking IPython notebooks and displaying them publicly on the web.

  • eLife is an open access biosciences journal which uses a non-standard peer review process in which the reviewers collaborate directly with each other.

There were also a lot of nifty groups and events:

  • Inspire9 is the little hackerspace in Richmond where the workshop was held, and they host regular Ruby and Python meetups, among other events. There's an entire room painted with flowers and butterflies, so I felt quite at home.

  • HealthHack in October this year, which brings together programmers and scientists to help solve medical research problems. It's been a good many years since I last used my biology background so I'm looking forward to this one.

  • The Open Knowledge Foundation runs HealthHack and lots of other cool stuff, like GovHack.

  • OpenTechSchool runs a bunch of experimental tech education projects, including Rails Girls which I'd heard about earlier. It has a Melbourne chapter!

I want to get involved in more of these things!