C++, Unicode and localisation

I’ve been dealing with some issues with Unicode at the moment, and trawling the internets looking for answers has revealed to me just how many people don’t seem to have a comprehension of what’s really going on in the string classes they’re using. I was one of them.


The first thing you need to read – before going any further – is Joel Spotsky’s blog on this:

The Absolute Minimum Every Software Developer Must Know About Unicode and Character Sets

If you haven’t read that, go there now. Right now. I’m not kidding, stop reading this and go read that. Come back afterwards. Nowait, don’t start reading more Joel Spotsky – come back! Oh, there you are.


Okay, so that was a simplified discussion about the history of text encodings and what it means to most developers. And it ended with “might as well use std::wstring, it’s native”. Well, here’s the kicker: not necessarily.


C++ is fantastic because it’s cross-platform. It can compile for Windows, Unix, OSX, iPhone, Android, DS, Playstation – you name it, if it supports native code there’s probably a C++ compiler for it. Unfortunately, not every C++ compiler treats everything the same. There’s a lot of ‘holes’ in the specification that let the compiler decide what it wants to do with certain data types, how big it wants to make them (e.g. on a 32-bit processor, the int datatype is 32 bits. On a 64-bit processor? You guessed it, 64. There needs to be room for compilers to set the size that works best on the native hardware and operating system. But this leads to issues when you want one piece of code that compiles to all platforms.


On Windows a std::string uses a single byte, which essentially means it can only support the first part of a UTF-8 encoding. On Unix a std::string is 2-bytes, and can happily use a UTF-16 or UCS-2 encoding. On Windows you need to use a std::wstring (wide-string) to have any Unicode-based foreign language support. On Unix, a wide-string uses 4-bytes, which can lead to strings being memory hogs and requiring conversion to and from the strings used natively by the operating system.


There are basically three ways around this:

  • use the time-honoured tradition of selective #defines and macros to compile using std::string for some systems, std::wstring for others etc. The Microsoft header <tchar.h> does something similar (using char or wchar_t arrays instead of STL classes)
  • define your own basic_string<uint16_t> template or similar that always uses the same size for a character. String literals become harder to use then, and need a macro-hack to work properly. C++0x (if it ever gets ratified), will introduce new Unicode support to make this method much easier. Interoperability with other APIs or libraries can easily lead to issues though.
  • go ahead and do what Joel Spotsky says: just use std::wstring and allow it to be UTF-16/UCS-2 on one system and UTF-32 on another. So long as you want your game running on a system and not sending text between systems/servers etc. this should be fine.

Personally I prefer the second option. I use string literals a lot less than I use everything else, and when writing Objective-C I have to prefix my string literals with an @ anyway, so wrapping them in a macro isn’t an issue for me. Plus it makes it easier to move forward with the next version of C++. Until that gets released though, hopefully you now know enough about Unicode to decide how to go about supporting (or not supporting) it in your next project.

The power of TODO

Lately my latest project has been slowing down. I haven’t been getting as much done on it as I was when I first started out. I had a few theories for why that was: partly because I’d spent the last month moving out form my apartment in Sydney, getting rid of all my stuff, living out of a suitcase, moving to Vancouver, and trying to find an apartment in Vancouver; partly because the core gameplay is done – and it’s fun – but now I need to add timers, high scores, menus, particles systems etc. to move it from feeling like a prototype to feeling like an actual game; partly because I’ve had new ideas since this project started and despite writing them down and thinking “I’ll look at them after this project”, there’s a certain latent excitement I have for trying these new ideas as quickly as possible.

But it wasn’t until last night, and a fantastic speech by Shane Neville at the October Vancouver Indie Game Developers Meetup, that I realised what the greatest influence on my work had been: I’d stopped writing TODO lists.


At the beginning of the project, my excitement drove both my passion and my organisation: I wrote code, design documents, sketched concept art and created Milestones. Over the course of the project though, as I hit (or slightly missed) those milestones and completed tasks, I’d been forgetting to create new ones as the scope of the project evolved. Gradually, and subtly, my task list had moved from pen and paper (and Trac and Wiki versions) to being stored in my head. Which is, without a doubt, the worst place to store anything important.


It simply doesn’t work to be keeping track of the important changes in my design in my own head, without anything written down. Thanks to that prompting speech from Shane, and a renewed excitement after meeting all the fantastic people in the Vancouver Indie games scene, my TODO lists have been placed back into their rightful physical form and the work on my project is progressing at a much greater pace as a result.

Conversations in Interaction

A recent blog post got me thinking about the essence of conversation, and how it manifests itself in games and other interactive media. The concept is that a book, or a movie, is a conversation between the writer and/or the director and the audience.


The big difference with interactive media is in the name: interactive. The ability to dynamically respond to the audience allows for a much greater, more in depth conversation to be had. The sad truth is that it rarely is: options are usually limited to the choose-your-own-adventure style of storytelling. To me this is like having a conversation with someone whose response, every single time, is to play a round of “would you rather”. “Would you rather be a sadistic, granny-hating killer or the world’s biggest mommy’s boy?” “Would you rather kill this character who wronged you in some way, or let him live despite the fact he’ll probably come back for you?” Although it’s been said many times before that most games lack a true greyness to their morality options, I can’t help but feel like this is a symptom of a larger problem rather than the problem itself: many games don’t create good conversation.


Conversation, in my experience, has a lot of different aspects to it. When meeting someone new, some of the most fun parts of a conversation are when you discover something in common (“Oh my god, I read that book too, when I was like 10!”). The interactive equivalent for that is finding things that both the player and designer thought of doing in the world, like typing “LHC” in Scribblenauts and spawning your very own Large Hadron Collider (and in fact, pretty much every other random thing you can spawn in that game). But usually we have to ask questions, in order to discover these commonalities. And one of the big differences between feeling involved in a conversation, and feeling like you’re watching someone else talk, is listening.


In my experience, people are willing to talk for longer than I expect when I don’t try to interrupt them. In fact, there’s like a shell: people start answering a question superficially; but when they’re given the time and interest to expand they truly talk about themselves, explaining links and concepts through their personal stories and ideals. Their eyes dart about in reflection of the awesome process of memory, creation and language that takes place in their mind as the conversation truly becomes about them. It truly is amazing to watch. Yet so many games fail to take the time to really let us put this kind of brain power into our interaction.


Game worlds are themselves massive and complex, as anyone who has – like myself – tried to build one knows far too well. The interactive equivalent of asking a player a question and then giving them time to answer is limited by just how creative they can be in the world around them. Every little piece of interactivity that could be possible needs to be manually designed, created, tweaked, tested etc. The way in which a player can be truly creative in their answers can often feel so limited by just how large or complex the team can possibly create the game world. But a game I’ve recently been playing – Minecraft – has me thinking another way. By limiting itself to ‘bad’ graphics and boxy, blocky elements (everything in the game world is a block, and blocks can be transformed into other blocks or special resources through crafting) Minecraft gives players the freedom to use their own imagination to fill in the gaps. It’s a technique which has fostered cult-status in games since the early days of text-based adventures – leaving space in the game world for the imagination to fill – but in Minecraft it’s really the essence of waiting for an in-depth answer to a question: what would you do in this situation?


In Sun Tzu’s Art of War, there’s a particular passage that comes to mind:

8. There are not more than five primary colors (blue, yellow, red, white, and black), yet in combination they produce more hues than can ever been seen.

The same can be said of elements in interactive media: it’s not about how many there are, but how many ways they can be combined that truly creates the space for players to exercise the creativity they each have.

A recommendation

Now, I wouldn’t normally do this in a blog post. I’ve recently moved from Sydney, Australia to Vancouver, Canada in search of some gainful employment at a games studio, but having been here barely more than a week I’ve been enjoying the touristy side of being in a brand new city and haven’t done as much programming as I would normally do. So instead of an in-depth discussion about a new programming technique, or a musing on how things are vaguely related, I’d like to take a moment just to make mention of some of the people that have been most helpful to me in all the craziness of relocation.


My friends, family and my girl backed me up the entire way, and I want to thank all of them for being so supportive. It’s not every day that someone you’re very close to decides they want to move very far away, and yet not one of them tried to get me to seriously rethink it. Having that kind of support really means a lot.


But there was another group of people who supported me. Not because they needed to, or because they were close to me, or even because they were required to by law! They seemed to help me just because they were nice people, and trustworthy professionals. Which I don’t think is something I ever would have said about a Real Estate agency before!!


Bentons has been a real estate in Annandale, Sydney since 1907, when it was first opened by J.F. Benton. It’s a business with a true legacy, being handed from Benton to W.J. Johnston down to A.F. Brown through the years, and finally to Terry and Shane Torpy – who have taken the dedicated work and outstanding reputation of the Bentons name into the 21st century.


My history with real estate agents previous to Bentons had been sketchy, but surprisingly typical for an Aussie tenant. One agent had tried to illegally hold bond and locked us out of the house because the windows were streaky after cleaning. Another wouldn’t send a plumber to unclog my drain until I arranged a complaint hearing with the Tenancy tribunal. Many renters are unaware of their rights and Real Estate agents take advantage of that to squeeze a little extra money for themselves, or to avoid having to do work they’d rather avoid. And it reflects poorly on the estate agents, on the profession as a whole, and on anyone that chooses to use such an agent to handle their properties.


The exception I’ve come across is Bentons. Not only do they know and protect the rights of both tenants and landlords, they bring the profession back to what it truly is: finding the right place for someone to live. It’s rare to find a salesman, or a manager, that truly wants the best for their customers. Shane Torpy and the team at Bentons are that rare breed. Despite some crazy issues with my housemate (I ended up having to chase her removalist down in Brazil to get her keys back. Long story), they were very understanding and released my bond to me the day I got the keys back. I’ve never known any other real estate to be that professional.  Ever. For a guy dealing with a lot of moving out issues and worrying about overseas money on top of that, getting my bond back on time was a godsend that took a whole lot of pressure off me for the last week or so in Australia.


So thank you Bentons. Even though you didn’t have to, you helped me more than you know. So I’m writing this blog to help you back :)

Renewed Desire

As I walked out of my last job and headed for the bus stop – my last day ended and a whole new life in another country ready to begin1 – I pulled my HTC Desire out of my pocket to check the time and it leapt out of my hands into a painful-sounding faceplant on the concrete, cracking the glass in front of the touchscreen. Devastated – and with glass splinters in my thumb from trying to check when the next bus was – I headed home.The broken phone.  And my broken dreams


A search on the Googles revealed that others in my situation had been quoted between $200 and $300 AUD to repair the touchscreen. Still others had chosen to ship parts from overseas repairers ($100 – $200 AUD) and replace the screen themselves. By the time I did my damage the phone was popular enough, and had been on the market long enough for me to be able to ship replacement parts from Hong-Kong for little more than $40 AUD. It turned out that the LCD was still working fine, but the broken glass cover at the front of the screen is actually the digitizer: The part of the screen that detects and sends the touches to the phone. 3 weeks later my replacement digitizer arrived and I set to work repairing my HTC Desire, following the instructions I found on this helpful website.


The screen came with several free tools for opening the phone (that didn't fit) A roll of double-sided tape and a $1 knife to attach the new screen

Tools needed:

  • – New digitizer
  • – Star screwdriver
  • – Cross screwdriver
  • – Separator
  • – Knife
  • – Double-sided tape

Step 1Step 1 is pretty easy – remove the cover, battery, SIM card and SDCard








Step 2 Step 2 is where I discovered that the cross-head screwdriver included with the replacement digitizer kit didn’t actually fit my phone, and had to go out and buy a new one. Note the VOID stickers that need to be removed to get at the screws: this procedure will end your warranty
Note the VOID stickers that need to be removed to get at the screws: this procedure will end your warranty



Step 3

Step 3 is where it starts to get hairy – try not to lose the Power button or Volume rocker as the circuit board comes out, and remember to unplug the antenna connection and to plug it back in when closing the phone up again.






Step 4

Step 4 gets a little scary- the screen is held into the frame pretty firmly, and it requires peeling and slight bending to get it out (not too much bending, mind you). There’s some metal tape stuff that has to be peeled off as well, but it’s pretty easy to stick it back on once you put the screen back in.





Step 5

Step 5 is similar to the last step, but worse. The digitizer is held in by a sticky black glue, and it’s necessary to pull it away far enough that you can cut it with the separator all around. I managed to do this without bending the frame, everyone else may not be so lucky.





There's a whole lot of gunk on the inside of the frame once the digitizer has been removed Easily cleaned off using the knife and some cheap alcohol (yes, Vodka works)

Step 6 just involves cleaning the frame so that the new screen can go in.






Step ??

Step 7: Put it all back together and…it doesn’t work. Crap.

Whenever I touch the screen it brings now the notification panel. I was lucky to get the phone unlocked for the picture.


Strangely, it seems to work fine when the screen is left just slightly out of the frame, so I went ahead with the next step. This eventually ended okay, as the tape made the screen work for some reason.

 Double-sided tape Cutting the roll into 2mm pieces

Step 8 is to glue the digitizer back into the frame using small (2mm) strips of double-sided tape placed along the edges.





Step 9 is to gloat!  I replaced my screen for less than $50!









Overall it was a surprisingly easy procedure, despite the worry about the screen not working about halfway through. If you can’t afford to get the screen replaced by an official repair centre, or don’t have warranty anymore, and you’re comfortable pulling the pieces out of a Tandy Robby Robot toy then by all means go ahead and replace the digitizer yourself!

Just don’t sue me if you stuff it up.

  1. I’ve since moved to Vancouver, Canada for a 2 year working holiday []

Multiplayer Madness

So I was recently linked to a youtube video, a tragic one for those of us who are quite attached to our consoles: Asian Girlfriend Destroys PS3.  After watching the video, I realised two things: firstly, the guy probably deserved what he got; secondly, there is a whole slew of similar videos in the related tab:  Hot Chick Smashes Boyfriends XBox 360 Console, Psycho Girlfriend Smashes XBox and they got me thinking about videos like Girlfriend Deletes WOW Characters.

Really there is no need to watch any of these videos, anyone can get the gist of them from their names. And it pointed out to me one of the intrinsic issues that I have with multiplayer games, and one of the main reasons why I don’t play them: there’s no pause button. In a single-player game, the whole thing is being run by you. If something important comes up, a friend pops around, or the kettle is boiled then it’s no issue to stop the game and do whatever it is that needs to be done: make a cup of tea, converse with your friend, do that vacuuming you said you’d do 3 days ago. This could mean pausing the game, or I can even switch my console off and know that I can always resume the story at the last checkpoint. Multiplayer games don’t have that luxury.

One of the things it’s easy to forget when my career is based around video games is that consumers have this thing called Real Life as well. And frankly, that’s more important to them than my game is – whether they treat it that way or not. I recently had a long discussion about the ethics of games like World of Warcraft that seem to deliberately offer incentives for remaining in the game (unless you’re in Korea, where they boot you out every so often to eat food). My argument at the time was that developers try to make their game as fun as possible, which can inadvertedly lead to mechanics not dissimilar to those of poker/slot machines and create an addictive experience. What I forgot at the time was the staying power intrinsic in being online with other people. See, most people are FOMO’s.

FOMO is the Fear Of Missing Out, and it happens to essentially anyone who’s leaving a situation where everyone else is going to stay. In a singleplayer game this feeling doesn’t often come up, because I can always go back and do that thing, or play through the game again differently. In multiplayer there’s a constant FOMO – I could miss out on experience points, sweet drops, that stupid thing Leroy did, killing that annoying guy (boom, headshot) etc. And now that there’s plans to punish rage-quitters there’s actually negative reinforcement around the idea of leaving. But how can you tell the difference between a rage-quitter and a guy who wants a cup of tea with his girlfriend?

This is I suppose a post of solidarity with the other bloggers out there who are criticising game length as a measurement of quality: Mutliplayer games are repetitive and essentially as long as the player’s attention span is. That doesn’t necessarily mean they’re good, just that people don’t want to leave them. Give me a 6-hr long epic single-player over a multiplayer mode any day of the week.

Other blogs on Game Length:


Just a note to point out (as most people have probably noticed) that I’ve finally made the oft-talked-about switch to WordPress hosted on my own site. A new blog post will be up tomorrow.

They’re everywhere…

I’m pretty new to iPhone development.

I mean, I’ve only been using Objective-C as a language since January this year. Almost a decade of C++ experience before certainly helped me pick up a few concepts, but I’m certainly not about to go parading myself as a guru anytime soon. In fact, I still end up turning to Google on a daily basis for insight on whatever issue I’m having with Cocoa Touch that hour.

Something I can’t help but make notice of are the number of accepted solutions out there that are bad. Not wrong. Most every time I find a solution it does indeed solve the problem at hand. But not elegantly.  Not efficiently. Not well written, not well named, not well tested. And really, just not good code. The examples that spring to mind include C++ code that marked int and float parameters as const in a member function (these integral types are copied when pushed onto the stack for a function call – they’re the same size as a pointer anyway.  Marking them const isn’t const-nazi power, it’s just redundant) and some Objective-C code that didn’t bother naming the second parameter of a method (Obj-C has a quirky style of naming any parameters after the first one, leading to highly descriptive method names like

middle:@"is" suffix:@"SPARTA!"

Not using this convention shows a deep misunderstanding of the language).
Any new or learning programmer that finds a solution of this sort and then tries to copy-paste said code into their project will be met with a rather unpleasant surprise. Their code instantly becomes less consistent, less standardised and less maintainable. Let’s be honest though – they deserve it.

Yes, I bet you thought this post would be a rant about how “solutions never work when I copy-paste them into my DetailViewController.m”. Not so, and shame on you sir for thinking so! For you see, I like it that Google’s proferred solutions don’t work like cookie-cutter magic. For me, as I expect it is with most people, if all I do is copy and paste then I don’t learn anything. If the code works without me having to do anything then the next time I get faced with a similar challenge my first thought isn’t going to be “I’ve solved this before”, it’s going to be “where was that website that solved this for me?”. And frankly, those people I’ve seen that do try to copy-paste the code and then comment asking for it to be slightly modified to suit their needs usually deserve whatever issues they bring upon themselves. And maybe it will eventually help them: I know from my years of programming that one of the best motivators for writing good code is having to maintain bad code in another system.

So bring on the bad solutions – seriously. As a blogger I’m usually tempted to try and make the code I release here robust and reusable and free of coupling issues. I don’t plan to stop that – I like to think that keepiing these things in mind are indicative or where I’m at as a programmer, these are the worries I’ve earned and I’m not going to drop them so that someone else can learn from mistakes I shouldn’t have made. But I’m not going to spend extra time to make that code as pretty and useful as possible: it’s up to the person that wants to use it to get in there and change every line and variable name that they need to – and learn what the code actually does at the same time.

Major misunderstanding

I made the discovery this week that the vector math library I had written for my current project wasn’t right: I needed to transpose any matrix I got from an OpenGL glGet…  call before I could use it properly,  otherwise my matrix-vector and matrix-matrix operations would just return junk values.  A quick Google search revealed a pretty big point of confusion around the OpenGL spec: column-major or row-major matrix representation? Which as it turns out is two points: how you store your matrices and how you represent your vectors.

There are two schools of thought on how to treat vectors when doing matrix-multiplication: traditional math treats them as columns in a 1-dimensional matrix, like in Figure 1.

Computer science, on the other hand, has tended to write them as a row, like in figure 2. I assume the difference comes about because the first method is an absolute bitch to try and type, but I’m only guessing. Anyway, the confusion comes about because the OpenGL spec is written using vectors-as-columns notation, in an attempt to bring us unruly computer scientists in line with the rest of mathematics. The way in which the code for a matrix-vector operation is written will be different depending on whether a vector is treated as a row or as a column.

The second point is how to store a 4 by 4 matrix in a contiguous bit of memory: to map it columns-by-column, like in figure 3; or store each row in order, like in figure 4. Obviously the position of data in memory will also affect how the actual code to perform vector-matrix operations is written.

So, as it turns out there was an issue with backwards compatibility when writing the OpenGL spec: they wanted to use vectors-as-columns, but to do so would break compatibility with the previous GL, which treated vectors as rows. So they did a sneaky on us. They wrote into the spec that OpenGL was column-major. The thing with column-major and row-major matrices is that you can switch between them by transposing i.e. a column-major translation matrix can become the same matrix in a row-major system by flipping its values along the diagonal like in figure 5.

And the thing with treating vectors as columns, is that you can switch between them by transposing i.e. in a vectors-as-rows system, you could treat vectors as columns by using the transpose of the matrix you’re multiplying against. See figure 6 (at which time I realised that Open Office comes with a perfectly good math editor).

So by taking the existing vectors-as-rows system, with row-major storage, and making it a vectors-as-columns system with column-major storage, OpenGL leaves us with the transpose of the transpose: the exact same matrix we started with!

Moral of the story: if implementing a vectors-as-rows math library is your thing – as it is mine for previously metnioned typing issues – and storing matrices row-first comes natural: keep doing both! If vectors-as-columns is your secret perversion, make sure you implement it with a healthy dose of column-major matrix storage or people will look at you weird. But no matter what you do, don’t waste almost a week writing and rewriting your matrix code to try to get it to play nice with OpenGL – read this article again instead :)