So, no blog post last week, I had actually managed to get some things done, just not really enough to blog about, my attention was irratic, jumping from task to task so much that I didn’t really get anything in a fit state to talk about. This week however, I managed to finish off some of the things I started last week, or at-least get them to the stage where I can write about them.

So the most major thing I did is that I finally moved the project over to using Premake, I’ve wrote about Premake before, but basically, it allows you to define project file that can then be compiled into Visual Studio, XCode or makefile projects (and more) so that you only have to maintain one file, (in my case, one file per-project and a master-file for the solution) and you can build on the 3 major platforms.

I also added the repository to Cerberus, (and it pretty much built first-time) so whenever I push to the main repository (hosted on BitBucket), Cerberus will download and compile the project on all the platforms I’ve set-up (currently only Windows, as I’m in the middle of setting up a Linux VM on my laptop) and e-mail me the result, it’s kinda cool to see my own CI software compiling my own engine, and then e-mailing me telling me it’s ok.

I’m anticipating some issues when I attempt to get Tyr compiling on Linux for the first time, I got a bit sloppy blocking off my Win32 specific code and I’m sure I will have fallen-foul of some of the warnings GCC has enabled that VS doesn’t, so I want to try and get this working as soon as possible, so I can sort these issues before I get too far with things. But I’m also tempted to leave it till the new year, where I shall be constructing a dedicated build machine or two (ideally one per-platform, but I’m debating just getting some OSX compatible hardware and VMing everything).

I also started the input library, (thinking I’d need it to progress with the editor) which saw me referring to this article by Keith Judge, which gave me a start diving into the weird and wonderful world of the Win32 RawInput API (I’d really love to talk to someone at MS who could attempt to justify some of those decisions…) which seems to be the way to go input-wise, and fits in with my whole taking care of things myself aspect.

Finally I looked into the editor, more precisely making it so you can translate, rotate and scale nodes using the mouse, like one would expect from a civilised program. I’d actually been working on that for a few days but apparently I wasn’t very awake when I was, it got to the stage where it was nearly working but not quite, but then the next day I had a look and it was glaringly obvious that the approach I was taking was making it stupidly hard for-myself and I re-wrote the thing in about half the lines of code of the original solution, and it worked just fine. So now you can drag nodes around the level… As long as those nodes have a size greater than zero, I’m attempting to come up with a solution for selecting/transforming nodes with no size in a way that won’t pollute the game-side code and only be available for the editor.

So that’s it for the past two weeks, this week I hope to clear up how I’m accessing properties (I’m planning some form of compile-time hash so grabbing properties by string is ok) and finish off the editor transforming code and add the buttons to the editor. Then I really need to look at dragging/dropping nodes from one part of the scene-graph tree control to another, as it currently very much doesn’t work but is a rather vital operation.

Recently I was reading an old post by that guru of good software thinking, Joel Spolsky (Yes you should know him, you almost certainly know something he’s done) in it, as part of a series on writing good software specifications, he explains the virtues of having a section on ‘Nongoals’, which are, quite simply things that you will not do making (this version) of the software. This is very useful, especially for smaller projects that have the tendency to quickly expand beyond the abilities of the team working on them.

I’ve been fighting against this feature/scope creep with my latest project, Cerberus. As I develop it, I keep thinking of cool features that I could add to make it better, but I’m trying to keep it in a scope that is manageable by me in a sane time-frame. Now most of these features make it a more general product, something that would expand it from it’s initial me-centric design to something that could appeal to more people, picking and choosing which of these features should make it into the project requires both discipline to resist the It would be so cool if it did this! feeling and foresight to evaluate if a feature may be of great use to me in the future, a third factor also comes into play, since this is a hobby project designed to teach me new things (Python and jQuery so far!) I also have to weigh-up how much of an opportunity for learning a feature provides.

An example of how I have expanded the scope of Cerberus is the addition of ‘build steps’. Initially, Cerberus was designed to do 3 main tasks:

1. Clone/update a Mercurial repo.
2. Run the Premake script.
3. Compile the generated solution.

This is a very me-centric design, all the projects I currently work on are hosted on Mercurial repos and they all use Premake to generate the solution files. Now as I was developing Cerberus, the idea struck me that it would be quite cool if, for each repository, you could specify a series of steps to perform on the contents of that repository, so for example, if the project was hosted on a Git repository, you can replace the ‘Mercurial’ script for the ‘Git’ script and you’re good, or say a particular project had a set of tests that should be run, just add a new build step and you’re good to go.

Initially I resisted this idea, thinking it was a brazen case of feature creep, I was building a tool for me, and that was more than I needed. But the longer I thought about it, the more I thought how limited it was making the project, it only suited my needs now and would then have an influence on my practices for the future, I never want to be in the position where I feel resistant to implement a new thing because Cerberus would have to be re-written to accommodate it. Secondly, a few people have expressed interest in this project, which I have to admit adds weight to the argument of making it slightly more generally appealing.

So, today, while writing the code to add and edit repositories, I added the ability to specify build-steps per-repo, currently there is only 3, one to clone or update a Mercurial repo, one to run a Premake script, and one to actually compile it, but I am expecting to add more to it soon. I am happy that I decided to implement this feature, but it got me thinking about a list of things I most certainly am not interested in implementing, and thus I present (in no particular order) the list of Cerberus Nongoals:

• No effort will be made to make Cerberus work in any browser that is not Chrome.
• Cerberus will not work if you have disabled javascript.
• Cerberus will only support Windows, OSX and Linux (Ubuntu).
• Cerberus will assume that anyone using it is generally not evil.

These may take some explaining, firstly, the Chrome thing, Chrome is the browser I use, Cerberus as it is should run just fine in any standards compliant browser, but then again it may not, I don’t really care, if it bothers you, download the source and fix it.

Secondly, Cerberus makes use of javascript to make things easier, it does not have a fall-back.

Thirdly at the moment I only have need/want to test my code on Windows, Linux and OSX. Solaris, BSD, Haiku and other people, if you want it add it.

And finally, Cerberus was meant to be run on an internal network, it is not meant to be open to the general malicious public, and as such, it only performs basic error checking, it is assumed that anyone using it wants to compile some code and doesn’t want to break things, I have no real interest in attempting to patch all possible vulnerabilities in a system that essentially allows someone to run Python code on your server. Again, for all these points, if this is an issue for you, download the source and fix it, this is the beauty of open source.

Cerberus continues to progress towards a feature-complete beta state, and I am pretty excited to see it taking shape like this.

Have been continuing with Cerberus  this week and managed to reach a rather important milestone, at 1:40am on Thursday 2nd June, Cerberus successfully cloned, and then compiled, the contents of the LibCT hg repository on Windows.

This is a pretty big step but there’s still a fair bit of work to be done. The fron-end needs some work still, especially repository management and the build report formatting, also need to write the Linux and OSX build slaves (which will hopefully not take long, since most of the build-slave code is common to all platforms)

I’ll leave you with a video showing off what’s been done so far, including a demo of the actual build process (best viewed in fullscreen HD, so you can read the text).

I’ve been busily working on Cerberus this week, mainly working on the website that acts as the front-end to the system. The entire front-end uses files rather than a database to store the data, I was considering using a database, MySQL or similar, but I really like the idea of the entire build system existing under a single folder, so it’s simple to backup and move around.

This is the main screen for the Cerberus front-end, it contains a list of all the repositories that Cerberus knows about, each repository also list its’ URL, and if it has been built, a summary of the report from the most recent build (clicking on the summary will take you to the full build). Also there is buttons to build each repository, which are now blue (previously they were green) due t the need to add an ‘Add Repository’ button below the list that should be green.

This screen shows the detail of a single report, with error and warning logs for each operating system, and also shows the amount of time it took to compile for that platform.

Here is the list of reports for all the repositories, the reports are listed in time/date order, with the most recent at the top. Each of the reports also shows it’s summary, which is the total number of errors and warnings that occurred during that build.

This screen shows the status of all the build slaves, most prominent is the operating system that the slave is running on, it also shows the IP and port they are listening on. (I was hoping to show more than one build slave active, however for some reason when I came to take the screenshots this morning both the build slaves were managing to be reporting themselves bound to the same port, which caused only one build slave to show up on this screen).

Not so much left to do with the front-end now, need to add the ability to add/remove repositories, and then finally hook up the build button and then it’s back into the internals to make the build process as solid as possible and report errors in a meaningful way. It’s still a fun project to work on and I’m looking forward to having it working hopefully sometime soon.

So I have been continuing to work on Cerberus, mainly focusing on getting CerryPy and Jijna2 working together to provide the web interface (which seems to be working nicely). I also fired up Photoshop to craft a layout of what this web-interface will look like:

Cerberus showing a list of registered repositories

I wanted to go for a clean ‘app-like’ look with a nice amount of spacing and after several hours of staring at it I still liked it so I started cutting it up and got to work laying down the HTML (5, obviously) and CSS to make a site which replicated this image.

Other than that I’ve been setting up the virtual machines (VirtualBox is such a nice piece of software to use) which posed an interesting problem.

Virtual machines (referred to as guests) do not have direct access to the network interfaces of the host machine, instead the guests (by default, this can be changed) connect to a software router provided by VirtualBox which allows the guest OS to do such things as access the internet however, a process on the host machine will be unable to contact a process on the guest machine, which is obviously an issue as the build_master will be running on the host machine and some of the build_slaves will be running on the guest machines.

Now, it is possible to configure VirtualBox to forward certain ports to certain guest machines, which will allow a process on the host machine to talk to a process on the guest machine and visa-versa, however, this then runs into the issue that effectively these processes will be running on the same machine, and thus cannot bind their TCP server sockets to the same port. This issue can be handled, an attempt to bind a socket to a port that is already bound will result in an exception that can be caught, and allow us to try another port until we find one we can use.

This results in a different issue, now we have n build_slaves, each running on a different port and we don’t really know which ports they are listening on, making contacting them rather difficult. The solution to this is a system I have implemented a few times before. The build_master fires up a UDP port that is listening for broadcasts, build_slaves broadcast messages to that port stating the IP and port the TCP server is listening on and also the operating system they’re running on, the build_master maintains a list of ‘active’ build_slaves (a build_slave is considered active if the build_master has received a broadcast message from them in the last 60 seconds. Now when the build_master receives a command to build a repository it simply communicates that to all the build_slaves in it’s active list.

So that’s my progress so far, I hope to have time this weekend to get the web-interface up and running so that I can trigger a build directly from that, again I’ll report on progress when there is some.

I recently got my shiny new laptop (from PCSpecialist, would definitely recommended them) which has now replaced my long suffering former laptop as my main development machine. This leaves me with a ‘spare’, decently powerful laptop which is perfect for a project that I have been wanting to do for a while now: Create a 1-click build machine.

Basically I want a system that allows me, with a single click, to trigger a script (or scripts) that will clone/update a Mercurial repo, execute the Premake script to generate the solution/project/makefiles, compile the project run a post build step (unit-tests or the like) gather all the information (warnings/errors/compilation time) and stick it in a nice web-page for my viewing. Also this needs to run on Windows, Linux and OSX, running on the same machine (natively running Linux, with Windows and OSX virtual machines).

Sounds like a fun project right? And I really like the idea that in one click I can know that my code compiles on all the major operating systems (and that I’ve not forgotten to commit any needed files). Also I decided to make this system in Python, since I’ve been meaning to learn it and nothing helps me to learn a language more than making a project in it.

So, I hear-by present, Cerberus:

cerberus.py

This is the python code that acts as the server and the main point of entry to the system, you will be able to access this from a browser and see the build reports and trigger new builds, as-well as taking care of other admin stuff.

build_master.py

This script, called by cerberus.py is responsible for triggering the builds across all platforms, it then collates the result of these, makes a pretty report out of it, and saves it as a timestamped html file that can be accessed by cerberus.py

build_slave.py

This script is a permanently running TCP server, waiting for build_master.py to contact it with the url of a repository. (Note that initially there will only be the option to build, in the future there may be other commands these scripts can execute.) It is responsible for cloning/updating the local repo and reports the results of the build command back to build_master.py

build_slave_[linux|osx|windows].py

These scripts contain the platform specific code to actually generate and build the projects and parse the output from the compiler into a standard format which is passed back up to build_slave.py

This is the plan anyway, I’m not sure if it will work, I’ve not looked at other build systems, I’ve never touched Python before and I don’t actually know if I can get OSX to run on my laptop, but it’s going to be fun finding out.  I shall post my results when there is any.

For a while now, I have had a task to create a CMake (or the like) script to generate the files needed to compile LibCT for various different compilers. (This was a task originally scheduled for version 2, and got pushed back to version 2.1 as a ‘nice-to-have’ feature).

The reason for my reluctance with regards to completing this task has been just how complex CMake is, even after a brilliant tutorial by my good friend Jamie. Now, don’t get me wrong, I’m not one to shy away from complex tasks, it’s good to have a nice mini-project to sink a couple of weekends into, and learning CMake probably would have been valuable in the future. It was the un-needed complexity that bothered me, the longer I looked at it the more the feeling grew that There has to be a better way to do this, and as much as I love a challenge, I love efficiency more and the thought of so much (apparently un-needed) work did not appeal.

Also on my ToDo list for a while has been to check out a little project called Premake, after reading about it at that fountain of useful information, #AltDevBlogADay so today I finally got around to checking it out, and everything I learned about it just made it better and better.

After a few hours of reading around the documentation, there was no bad news, it supported everything I wanted to do, and in about half the number of lines it would take to get CMake to do things. It was also really rather fast, generating projects much more speedily that CMake had ever done.

I’ll let their (ample) documentation and sample-code explain the system better, but the up-shot of all this is that after a few hours of faffing around I’ve got a Premake script capable of generating (currently VS2010) projects that can successfully build both the LibCT library and EdCT executable.

Task Complete.

So, one of the good things about the recent Sony break-in is how it has brought security issues to the attention of more people, these days it’s not just about having one good password, because people may still get at it, you cannot trust sites to follow the best practices (which is currently to add a salt to the user submitted password, then hash that and store the hash, websites just storing the hash of your password are probably not doing enough), you really need a per-site password solution, which is difficult, I mean, most of us have probably 20+ different sites that we log-in to on one basis or another, and it’s difficult to come up with, let alone remember that many different passwords.

However, I am going to show a nice easy technique that will give you unique, per-site very strong passwords (as in, ‘would require the resources of a government to break’ kinda strong). Now, this method has been around a while and I can’t take any credit for coming up with it but after talking about it to a few people who didn’t know it I thought I’d share it there so that anyone reading this can make their online self safer.

Come up with a strong ‘base’ password

This should be long, and alpha (upper and lower) numeric and symbols, this is the core of your per-site password so make it as strong as you can, then burn it into your memory (you’ll be typing it a lot, so you’ll remember it eventually) mine is 15 characters. At a minimum I’d suggest 10 (the recent Gawker attack proved the benefit of longer passwords, it turned out that Gawker were only storing the hash of the first 8 chars of your password, so if your password is longer than that, the first 8 chars are useless on any other site).

Append 2 chars for each site

Something easy to remember, so ‘tw’ for twitter, ‘gm’ for Gmail, etc., this will give you a password of ‘<BASE_PASSWORD>tw’ for twitter, for example. Now for each site you only have to remember the two chars, and since they are logical and related to the site, it’s easy to remember (sometimes I forget, but my first guess is pretty much always correct because my mind comes to the same conclusion about what characters I should use as it did when I created the password).

This also adds an extra 2 chars to the length of your password (if you really want to be good, try capitalising one of the characters but not the other).

This will give you a password that is impossible* to crack, and importantly, a different one for each site, so the next time a site you use is compromised, you don’t need to worry about any of your other accounts.

(* Impossible in that it would be much, much cheaper to hire someone to put a gun to your head and make you tell your password)

Now, I understand to some/most people this sounds like overkill, and it probably is, no-one will care that much about any account I own, but for the small amount of effort it takes (ok, it may require some work to change all your existing accounts to this system but it’s an evenings work at most) it is, in my opinion, worth the piece of mind to know that I have done everything I can to protect my accounts, my (on-line)identity and the information about me and my friends that I have access to.

We all know we should backup our data, ideally the backup solution we use should be ‘set-and-forget’ because, honestly, who remembers to backup? The guy who’s just lost his data, that’s who, 3 months from then? Maybe, possible, occasionally, on that external that’s kicking around somewhere.

I got seriously into backing-up up my data in the final year of uni when both my PCs died in the same week (lucky for me I had enough time to bring one back up before the other died, so I could carry on my work).

Now, thankfully at the time I was backing up to an external drive pretty regular (I had a batch-file to take care of it) so I didn’t loose too much, but the experience convinced me that I wouldn’t mind paying for the piece of mind to know that I have a proper backup solution in place.

It was shortly after this, I realised how shockingly cheap on-line backup is.

My current backlup host is CrashPlan, chosen from a long, long list of competitors for the following features:

• Unlimited backup space
• Cross platform (Windows/Linux/OSX)
• Nice, easy to use client
• Encrypted backups
• Backup to a folder/external drive for free
• Backup to a friends/other computer for free
• Continuous backup (every-time I’m not doing anything, my pc is backing up)
• Adopt a computer (prevent a fresh full-backup if you format your pc (I’d get this for this reason alone))
• Cancel any-time and get any remaining contract time refunded
• E-mail/Twitter notifications, either a backup report showing success, or a warning if a backup hasn’t succeeded for a while

An impressive feature set, I think you’ll agree, and I currently get all this for £2.50 a month. ($4.17, so it depends on the exchange rate)

Yup, for the price of a sandwich you too can get all this, that price is based on a yearly contract (pretty much risk-free if you can afford the up-front cost, as you can cancel at any-time) you can get it even cheaper if you go for a multi-year contract.

Now, the issue with on-line backup is the time it takes for that first backup, the CrashPlan client is pretty fast, and it only took 3/4 days to send over the 25+ gig of data I backup (which isn’t too bad, but you may get better) which is why the ability to ‘adopt’ a computer is invaluable. I’ve just bought a new laptop (more on that later) and when it get’s delivered it will be my new main computer, so I’ll move my CrashPlan onto that, if not for the adopt feature, I’d have to re-do that initial upload (my original backup provider, Mozy (whom I don’t recommend) didn’t have this feature, and it was a pain to have to re-upload that much data). Now I’ll just have to click a button saying ‘yup, that data now belongs to this computer’ and all is well.

So, don’t wait till something bad happens to get serious about backing up your data, for this price, you really can’t afford not to.

This post is a continuation of the How I get things done, for a while I was stuck on my most recent project (an XNA game). Despite having a large number of tasks set out, in a priority order, my next task (na) was causing me issues:

‘New Game Editor’

This, as a task, sucks.

It sucks even worse than a task of ‘Make game now’, because at-least that one is obviously bad.

It seriously took me about a week of being stuck on this task before I realised that it was way too big and too vague, especially given the complexity of the map editor (as it intends to be user-facing), I didn’t know where to start with it.

So one night I decided to do what I should have done in the first place, break that task down into a number of different tasks (about 15) which encompassed the next few parts that will eventually make up the ‘New Game Editor’.

The next week, I stormed through these more manageable tasks, causing the editor to improve greatly (it’s kinda getting somewhat close to being complete… Ish)

If you ever find yourself staring at your ‘na’ task, wondering where to start it’s too big take that task and break it down further until it’s obvious how you should proceed.