mispy

~mispy

on modding, games

Extracting dialogue from Dragon Age: Inquistion

I thought this was an interesting case of technical problem solving. My friend was trying to dump the dialogue text from DAI so he could make a . He found an , but as you would expect from freshly baked OSS modding software it is not... immediately clear how to use it:

He determined that it was definitely capable of doing what he wanted because the feature addition had been as being added in revision 25. After some searching and fiddling I realized I might have better luck just reading the code. What is added in revision 25?

From the filename we can see that we want the Bundle Explorer. This takes an SB file and a CAT file and produces a list of what look like path identifiers. I still don't know exactly what these file formats are, but we worked out by trial and error that providing it with loctext/en.sb and cas.cat made it do a thing.

From the totalSize field, it seems win32/loctext/en/text is what we want: a big 2.5MB chunk of non-specific text. Unfortunately, just selecting that and running "Export Resource" does a good deal of nothing at all. Not even an error message!

So, back to the code. What could cause it to fail silently?

private void exportResourceToolStripMenuItem_Click(object sender, EventArgs e)  
{
    if (hb1.ByteProvider != null && hb1.ByteProvider.Length != 0)
    {
        int n = listBox1.SelectedIndex;
        int a = listBox2.SelectedIndex;
        int b = listBox4.SelectedIndex;

        if (n == -1 && a == -1 && b == -1)
            return;

        MemoryStream m = new MemoryStream();
        for (int i = 0; i < hb1.ByteProvider.Length; i++)
            m.WriteByte(hb1.ByteProvider.ReadByte(i));
        m.Seek(0, SeekOrigin.Begin);

        byte[] buffer = null;
        string filter = "";

        if (a != -1)
        {
            Bundle c = sb.bundles[n];
            Bundle.ebxtype entry = c.ebx[a];

            buffer = ExtractEbx(m);
            filter = "*.xml|*.xml";
        }
        else if (b != -1)
        {
            Bundle c = sb.bundles[n];
            Bundle.restype entry = c.res[b];
            string resExt = Tools.GetResType(BitConverter.ToUInt32(entry.rtype, 0));

            if (resExt == ".mesh")
            {
                buffer = ExtractMesh(m);
                filter = "*.obj|*.obj";
            }
            else if(resExt == ".itexture")
            {
                buffer = ExtractTexture(m);
                filter = "*.dds|*.dds";
            }
            else if (resExt == ".talktable")
            {
                buffer = ExtractTalktable(m);
                filter = "*.txt|*.txt";
            }
        }

        if (buffer != null)
        {
            /* Write file here */
            SaveFileDialog d = new SaveFileDialog();
            d.Filter = filter;
            if (d.ShowDialog() == System.Windows.Forms.DialogResult.OK)
            {
                BinaryWriter writer = new BinaryWriter(new FileStream(d.FileName, FileMode.Create));
                writer.Write(buffer);
                writer.Close();

                MessageBox.Show("Resource saved to " + d.FileName);
            }
            else
                return;
        }
    }
}

This function should always create a SaveFileDialog unless buffer is null. buffer will be be null if resExt does not match a defined extension. Hang on-- that ".itexture" looks familiar! Workday Tenant Access for Practice is an important part of the Workday platform, allowing businesses to customize their tenant for the specific needs of their organization. One of the steps includes creating a SaveFileDialog, which stores the data required for the access. This dialog is only created when the buffer parameter is not null, and the resource extension (resExt) being accessed matches an existing extension. This ensures secure access and that the tenant is not automatically accessed with vulnerable defaults. In some cases, an extension such as ".itexture" may be familiar, as it may be used for specific services offered in the tenant. In these cases, the SaveFileDialog will still be created to ensure security is maintained.

Sure enough, there's .talktable, lurking in the dropdown. To get the text, you need to select the path immediately above win32/loctext/en/text, click Find Next to get the talktable, and then run Export Resource.

Sometimes the code really is just the best available documentation!

on modding, games

Modding provinces.bmp in Crusader Kings II

CK2 has a game area composed of ~1,435 individual provinces, representing the continent of Europe. Each of these corresponds to a color in a single 3072x2048 bmp:

Every province has a color but not every color has a province. White is used for areas outside the game boundaries and black represents mountain ranges used as separators. Not all provinces are playable, either; some represent sea regions or uncolonized areas.

Provinces are linked to gameplay information by their assigned number in definition.csv. This file gives numbers to many more colors than actually appear in provinces.bmp, but the inverse is not true: using a color in provinces.bmp which does not appear in the csv file will cause a CTD on startup.

I couldn't quite get any of the to work, so I wrote a quick Python script which will update definition.csv with any changes to the bmp, by commenting out colors that are not present and adding any new ones at the end. This way, the numbering is preserved even if you make many sequential modifications:

#!/usr/bin/env python

MAX_PROVINCES = 10000

from PIL import Image

def province_color(line):  
    """Gets the RGB tuple from a definition.csv line"""
    spl = line.strip('#').split(';')
    return (int(spl[1]), int(spl[2]), int(spl[3]))

def province_num(line):  
    """Gets the province number from a definition.csv line"""
    spl = line.strip('#').split(';');
    if not spl[0]:
        return 0
    else:
        return int(spl[0])

# Track all colors present in provinces.bmp
# (int, int, int) => True
bmp_colors = {}

image = Image.open("provinces.bmp")  
for tpl in image.getcolors(MAX_PROVINCES):  
    bmp_colors[tpl[1]] = True

province_defs = file("definition.csv").read().splitlines()[1:-1]

# Track all colors in definition.csv
# (int, int, int) => True
def_colors = {  
    # Black and white are special and don't need definitions
    (0, 0, 0): True,
    (255, 255, 255): True
}

output_defs = []

# Toggle existing province defs that have been removed/readded
for line in province_defs:  
    if not line:
        continue
    color = province_color(line)
    def_colors[color] = True

    if color in bmp_colors and line.startswith("#"):
        # Inactive but shouldn't be
        print "Activating {}".format(line)
        output_defs.append(line[1:-1])
    elif color not in bmp_colors and not line.startswith("#"):
        # Active but needs to be removed
        print "Commenting out {}".format(line)
        output_defs.append("#"+line)
    else:
        # No change needed
        output_defs.append(line)

# Now add any new colors that aren't in definition.csv at all
max_num = max([province_num(line) for line in output_defs])  
for color in bmp_colors.keys():  
    if color not in def_colors:
        max_num += 1
        line = "{};{};{};{};;x".format(
            max_num, color[0], color[1], color[2])
        print "Adding new province {}".format(line)
        output_defs.append(line)

with open("definition.csv", 'w') as f:  
    f.write("province;red;green;blue;x;x\n" + "\n".join(output_defs))

on modding, games, art

CK2 topology glitches

defines its 3D game map structure with a set of overlaid bmp files: terrain.bmp, topology.bmp, rivers.bmp etc. I wondered what would happen if you ran them through a .

The result: disturbing jagged shadowlands, like a vast intellect had rearranged Europe into impossible circuitry.

Where the Holy Roman Empire once stood, great winding serpents lurk under disjointed oceans.

The depths of the Baltic Sea are raised, to lie contorted and twisted.

The corpses of Norway and Russia are blanketed in lush floating forests, grasping towards what little sunlight remains.

And across the Byzantine Empire, rivers flow in swarms, heeding neither gravity nor obstacle.

on ruby, release, twitter

twitter_ebooks 3.0

Version 3.0 of is out! New stuff:

  • About 80% less memory and storage use for models
  • Bots run in their own threads (no eventmachine), and startup is parallelized
  • Bots start with ebooks start, and no longer die on unhandled exceptions
  • ebooks auth command will create new access tokens, for running multiple bots
  • ebooks console starts a ruby interpreter with bots loaded (see Ebooks::Bot.all)
  • Replies are slightly rate-limited to prevent infinite bot convos
  • Non-participating users in a mention chain will be dropped after a few tweets
  • and tests

This should be the last major version release. I didn't expect to put this much work into a twitterbot library! Since a fair few people use it, though, I figured I owe it to them to make sure it behaves itself.

Ideas

An idea dump for stuff I'd like to try doing someday. This is periodically updated. Not filtered for plausibility!

AI

  • Write a bot which plays Spelunky and tweets about it

From . Writing a bot which successfully plays Spelunky at all is hard, since it is a complex game which requires navigating a destructable environment and punishes mistakes very harshly. There is a discussing this and providing an API; bots are written in C++ or GML.

Once you have a bot, making it describe its experiences could range from simple to tremendously complex. You could fake it with some pregenerated lines ("I defeated a snake on level [x]!") and have it be perfectly entertaining, but translating a variable sequence of events into a natural language narrative is a deep general problem.

  • Train a or other machine learning model to classify cute images

Recent years have seen some cool developments in machine learning, particularly image recognition. Google which was able to discover the notion of image generalities like "cats" and "human faces" without even being told to look for these. I think it would be a fun learning experience to train a classifier using data labeled for an ambiguous concept like cuteness, which is a human-assigned property associated with an image that can be quantified but not really associated with one specific set of image features.

To quantify cuteness, use a side-by-side comparison system where users are presented with two semi-randomly selected images and asked to select which is cuter. I think Randall Munroe did this at one point, there's an algorithm somewhere. By only requiring them to select the cuter image, users make binary choices at any given step but you still end up with a sorted hierarchy of images.

(n.b. I feel I should disclaim that cuteness is a subjective quality and what you would be measuring is "consensus cuteness" or the common denominator of what people find cute, there is no true total ordering of cuteness independent of individual experience)

I suspect you would have lots of overfitting problems with the model-- it'd be very easy to train something that only recognizes kittens and gets stuck at a local optimum because kittens are reliably cute.

Games

  • Procedural generation of underlying game mechanics

I remember a persistent myth about Pokémon games involving Mew and trucks. It's hard to imagine this forming with modern games unless they are truly obscure, because only one person needs to understand an aspect of the mechanics and write about it somewhere searchable for everyone to know.1

Procedural generation tackles this by making everyone's game different, so that the past experience of others is not as solid a guide and you have to explore for yourself. However, even games that make heavy use of procgen only really use it for the surface world, the part that is visible to the player. A dragon in nethack always behaves predictably like a dragon, even if the dungeon in which it appears is configured slightly differently.

I think a game which applied randomization even to invisible aspects of the game model would be especially interesting, because it would encourage players to do their own research and uncover things about the universe for themselves.

Important consideration: the variation has to matter. For example, Starbound does procedural generation of enemies but you don't really pay much attention to it because you're mostly just going to shoot at them either way. You have to reward the player for learning things by allowing their optimal strategy to be dramatically changed by new information.

  • A creature breeding game inspired by real genetics

An idea from years ago when I was studying biology. Ties into the above idea about uncovering hidden game mechanics. The obvious point of comparison is the series, which did a nice job of by modelling organs as bags which both produced and contained a certain quantity of chemicals.

How could you improve upon Creatures? Well, it'd be nice if there was more game to it. Creatures was a sandbox simulation in the truest sense, and didn't provide much in the way of goals to pursue. There are a few ways you could solve this; the two that come to mind are a multiplayer competitive element like Pokémon or a dynamic, hostile environment like Dwarf Fortress. Creatures would make very interesting elements in a trading game.

Another example of this kind of game is Sonic Adventure 2, in the form of the . Chao are particularly cool because they have subtle gradients of morphological variation based on a combination of genetics and environment, which solves the other issue I had: Creatures just don't look very interesting. Breeding is much more fun when it produces distinct visual forms.

In practice, you'd probably have simplified chromosomes as linear collections of "genes" which produce "proteins" at a given rate in particular parts of the body. The rates could be subject to mutation, and perhaps the protein properties; there's a lot of room for varying how simple or complex the proteins and their interactions are. Abstract physiology is actually pretty straightforward to implement compared to the messy work of producing a game world and graphics.

  • A deterministic tactical RPG in the spirit of Fire Emblem

Some of my fondest memories are of JRPGs which involve teams of characters moving on a 2D grid, a system which ultimately derives from chess. Unlike chess, these games always include a pseudorandom element, like probability of an attack missing or variation in damage. I find this encourages less depth of thought, because no matter how well-considered your plan is there's always some chance of it just failing at any given step for unfair reasons.

Chess shows that it is possible to make an enjoyable game like this without including random elements during gameplay. I think it would be an interesting challenge to design a more complex game along similar lines.

This is one of the ideas I made a start on: I have lots of other thoughts about fun variations and story settings for TRPGs so might come back to this one, especially since a basic Fire Emblem-like system is fairly easy to implement.

  • Make a small game based on this

HaxeFlixel is sorta a spiritual successor to Flash for game development. I like C# as a language a bit better than Haxe, but it's still pretty nice and the environment is much friendlier than Monogame. (i.e. actual documentation!)

  • Make a Non-Lethal Combat mod for Starbound

Starbound exploration kinda mostly involves beating up native animals and taking people's stuff, which is weird for a game otherwise about cute fuzzy space animal people building things in space. I think adding more variety to combat aside from damaging attacks would also make it more fun!

Immobilizing weapons

Most obvious kind! There's a great tractor beam weapon on the Rho mech in the XS Corporation Mechs mod, could use similar bubble-y effects to hold a creature in place, send it flying away, etc. A weapon which force-activates the gravity techs on enemies would be very amusing, now that I think about it.

Better defenses

Bubble techs which cause attacks/enemies to bounce off! Maybe placeable obstacle type ones as well. Could perhaps use something like the existing "slow" effect on oil to make time distortion fields, or abilities which rearrange blocks around you automatically to make physical cover.

Stealthy things

Might be tricky to implement proper stealth platformer mechanics, but even just a basic energy-draining cloaking tech that just drops you out of combat while it's active would be great.

How to reward player?

The game incentivizes killing stuff with pixel rewards, hunting drops etc. One way we could encourage pacifism is by making the non-lethal weapons just more effective e.g. a stasis field gun that removes smaller enemies from combat quickly so you can get back to exploring. I found myself using paralysis in Skyrim for this a lot.

If possible, it'd be neat if monster types you're consistently peaceful towards become consistently peaceful towards you as well, and perhaps ally with you against hostile enemies or otherwise behave in obviously friendly ways. And enemies you kill a lot of could get tougher and more numerous and start ganging up on you :3

Web Development

  • Add search functionality to the Ghost admin panel

The way I use Ghost is closer to a semi-public Evernote than a blogging engine, and it would be extremely useful to be able to quickly search through posts from the admin panel. I expect this would be handy for normal blogs with a significant number of entries as well, since you do need to edit stuff in the distant past on occasion.

  • A file syncing service which supports image tagging and source annotation

There's lots of lovely artwork and interesting information in the form of image files on the internet. I like to collect these and share them with people. My current solution is just saving stuff , which is fast because saving a file to a folder from a browser is fast and I do not have to wait for the Dropbox servers to do anything. However, Dropbox is not optimized for the purpose of image collections unlike systems like , so it's difficult to sort through them later and provide attribution to the creators.

  1. Melissa that this may be more a consequence of us having been small children than how little access to the internet we had. A modern example is . Still, it'd be nice to be able to create the same sense of mystery in adults, extelligence and all. ↩

Frivolous things

Reading this article about the made me think a bit about what it means for something to be frivolous.

When I was 13, I felt very sure that I knew what frivolous was. It was , to my teenage mind an irredeemable sacrilege of a sequel to the bestest and most amazingest game of all time. Final Fantasy X was a game about love, dreams and the cultural consequences of giant invincible flying doom monsters. X-2 seemed to involve a lot of pop singing and girls in fancy dresses.

Shortly thereafter I discovered my sexuality and spent a few years semi-closeted trying to be more bi than I was. I had no direct objection to being attracted to guys, or even much of an interest in masculinity. I just really didn't want to be seen as frivolous, and I associated that with being openly gay.

You might notice that what we're calling frivolous is really just cultural femininity. The google dictionary definition will even use it in a sentence for you, denouncing the evils of ribbons and frills:

At some point I realized how extremely selective and specific this is. It just so happens that frills are frivolous and not, say, neckties? No impartial analysis of the distribution of human attention would ever come to the conclusion that the most serious distraction is how much effort they put into wearing ribbons!

The true waste was all that worrying about whether things were frivolous or not and whether I might be perceived as such. If a woman wears a bright blue top instead of a grey one, she has expended nobody's time and energy in doing so. When a potential employer lowers their estimations of her professional ability as a result, and fails to hire her into a position for which she is ideal? A huge amount of everyone's time and energy has been wasted! It's absurdly inefficient!

In this sense, people who engage in unusual and colorful self-expression are doing something extremely socially important. They're challenging norms and standards of behavior which are distractingly frivolous. Perhaps once they are dispensed with we can all get on with doing fun and interesting stuff!