Unity — Devblog — Lavapotion

When we started our project at Lavapotion, we wanted an automated way to build our game to any platform we wanted as well as a seamless and easy way to give these builds to ourselves, stakeholders and testers.

With the number of great services that is available today, we decided to use Unity Cloud Build to build and then use a script that uploads all the successful ones to HockeyApp with the commit messages as the changelog.

Why Unity Cloud Build?

Unity Cloud Build got options to unit test, build and export to platforms like iOS, Android, MacOS and Windows. Which is all we need at the moment. They also have the options of choosing exactly what Unity Version to build from if you want, and they always have the latest as an option to.

It's super easy to connect with slack (for build notifications) and the setup is pretty straightforward. Disregarding that sometimessome builds fail for no reason, it's really everything we need at the moment.

Why HockeyApp?

Sure, you can share all your builds in Unity Cloud Build by creating a share link and send it to the people that needs it, but there's a whole manual process involved there and no easy way to restrict downloads to certain groups.

HockeyApp got distribution groups that you can add/restrict to any builds pretty much whenever you want. You can also notify people with an email with a changelog when a new build is ready. It's neat.

Connecting the two platforms. Why do we need it?

The obvious way to connect the services is to do it manually everytime. To break it down into a step-by-step instruction - take a look at the list below.

Log into Unity Cloud Build
Download all the successful builds for the platforms you want to upload to HockeyApp to your computer
Log into HockeyApp
Choose the app that's on the platform you want to update.
Upload a new version
Write/Copy-Paste the changelog
Choose the Distribution Groups that should have access to this specific build.
Decide if they should get an email notification.
Repeat steps 4-8 for every platform you would like to update.

A philosophy that we strongly hold at Lavapotion is to reduce all the "screw-up-able" steps that can occur in our process and automate as much as possible. Not only is the list above extremely time consuming, but almost every step is error prone to all kinds of mistakes because there are so many manual steps.

So as a first step, we decided to reduce the manual steps to the following when there's a successful build.

Run a script.

That looks way better, doesn't it? ;)

Setting everything up

You will need to do the following before we start:

Your project in a Git repository (for the automatic changelog)
Install Node Package Manager on your computer
Setup your project in Unity Cloud Build
Setup your applications in HockeyApp

The script we're creating will be written in Javascript and using a gulp job to perform the actual tasks of downloading and uploading, so we will need to setup the node project first.

Create a file called package.json at the root of your project and fill it with the following:

(Replace the placeholder with your own information.)

{
  "name": "Super secret project",
  "description": "The best super secret game in the world.",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "gulp": "node ./node_modules/gulp/bin/gulp.js"
  },
  "repository": {
    "type": "git",
    "url": "<YOUR GIT URL>"
  },
  "author": "<YOUR COMPANY NAME>",
  "license": "<YOUR LICENSE>",
  "private": true,
  "devDependencies": {
    "gulp": "^3.9.1",
    "gulp-download": "0.0.1",
    "gulp-hockeyapp": "^0.2.4",
    "gulp-rename": "^1.2.2",
    "request": "^2.80.0",
    "simple-git": "^1.67.0"
  }
}

This will setup a simple npm project. The devDependencies object explains what modules we will use in our scripts. The "gulp" part in scripts is optional - it's added so that we can easily use the locally installed version of gulp rather than a global one.

To install all of the devDependencies, open the command prompt (or terminal) and locate yourself at the root of the project and run the following command:

D:\YOUR_PROJECT_PATH>: npm install

After everything is installed, you can create a file called Gulpfile.js in your project root.

Gulp!

Given that your project is setup in Unity Cloud Build and HockeyApp, you can setup your Gulpfile.js like this and change the placeholders to your specific settings. This example will use an android build to upload.

'use strict'
var gulp = require('gulp');
var gulpDownload = require('gulp-download');
var rename = require('gulp-rename');
var request = require('request');
var git = require('simple-git');
var hockeyApp = require('gulp-hockeyapp');

var cloudBuildSettings = {
    apiKey: "<YOUR_CLOUDBUILD_API_KEY>",
    organizationID: "<YOUR_CLOUDBUILD_ORGANIZATION_NAME>",
    projectName: "<YOUR_CLOUDBUILD_PROJECT_NAME>"
};

var hockeyAppSettings = {
    apiKey: "<YOUR_HOCKEYAPP_API_KEY>",
    androidDevKey: "<YOUR_HOCKEYAPP_APPLICATION_ID>",
    devTeamID: <YOUR_TEAM_ID>
};

var tasks = {
    hockeyApp: 'hockeyapp',
};

var paths = {
    dist: 'dist/'
}

Downloading information about the latest build

Before doing anything, we need to be able to download the build information of the last successful build on Unity Cloud Build so that we can download it to our computer.

Luckily, Unity got a REST API that we can use to do this that is well documented.

Using the "List all builds" method, we can fetch the information that we need. Add the following function to your Gulpfile.

function downloadBuildInformation(pBuildTargetID, pOnCompleteCallback) {
    var baseURL = 'https://build-api.cloud.unity3d.com/api/v1';
    var options = {
        url: baseURL + '/orgs/' + cloudBuildSettings.organizationID + '/projects/' + cloudBuildSettings.projectName + '/buildtargets/' + pBuildTargetID + '/builds?buildStatus=success',
        headers: {
            'Content-Type': 'application/json',
            'Authorization': 'Basic ' + cloudBuildSettings.apiKey
        }
    };
    request(options, function (error, response, body) {
        var latestBuild = null;
        if (!error) {
            var builds = JSON.parse(body);
            var latestDate = new Date(-8640000000000000);
            for (var i = 0; i < builds.length; ++i) {
                var currentBuild = builds[i];
                var finishedDate = new Date(currentBuild.finished);
                if (latestDate < finishedDate) {
                    latestBuild = currentBuild;
                    latestDate = finishedDate;
                }
            }
        }
        else {
            console.log('Failed to get build information! message: ' + error);
            pOnCompleteCallback();
        }
        if (typeof (pOnCompleteCallback) !== 'undefined' && pOnCompleteCallback != null) {
            pOnCompleteCallback(latestBuild);
        }
    })
}

This function will download the successful builds available and return the newest one in the complete handler once done.

Downloading the build

After we get the information, we need to download the build to our computer. The build will be downloaded with a npm module called gulp-module. When the file is downloaded, we will return the file name in the complete handler.

Add the following function to your Gulpfile:

function downloadLatestBuild(pBuildInfo, pOnCompleteCallback) {
    if (pBuildInfo === null) {
        throw 'Cannot download build! Information is null!'
    }
    var latestDownloadFileName = cloudBuildSettings.projectName + '_' + pBuildInfo.buildtargetid + '.' + pBuildInfo.links.download_primary.meta.type;
    gulpDownload(pBuildInfo.links.download_primary.href)
        .pipe(rename(latestDownloadFileName))
        .pipe(gulp.dest(paths.dist))
        .on('end', function () {
            console.log('\nFinished Downloading Build\n');
            pOnCompleteCallback(latestDownloadFileName);
        });
}

Creating the change log

Since HockeyApp supports uploading a change log in markdown, we wanted a quick look on what's been done since the last build. At this point we don't need any curated or filtered logs, so we can just use the git commit logs for now (given that our commit messages are properly written).

To do this, you can add the following three functions to your gulp file. One takes the information available from unity cloud build and the other two gets information from your git repository and retrieves the commit messages.

function createMarkdownReleaseNotes(pBuildInfo, pOnCompleteCallback) {
    if (pBuildInfo === null) {
        throw 'Cannot get changelog! Information is null!'
    }
    var markdownChangelog = '#' + cloudBuildSettings.projectName + ' ' + pBuildInfo.buildTargetName + '\n';
    markdownChangelog += '**Branch:** ' + pBuildInfo.scmBranch + '\n';
    markdownChangelog += '**SHA1:** ' + pBuildInfo.lastBuiltRevision + '\n\n';

    var ignoreCommitIds = [];
    if (pBuildInfo.changeset.length > 0) {
        markdownChangelog += '## Recent Changes \n\n';
        markdownChangelog += 'If installed, this build will: \n';
        for (var i = 0; i < pBuildInfo.changeset.length; ++i) {
            var message = pBuildInfo.changeset[i].message;
            markdownChangelog += '* ';
            markdownChangelog += message;
            markdownChangelog += '\n';
            ignoreCommitIds.push(pBuildInfo.changeset[i].commitId);
        }
    }

    var gitLogSettings = [];
    var maxHistoryLogs = 50;
    gitLogSettings.push(pBuildInfo.lastBuiltRevision);
    gitLogSettings.push(pBuildInfo.scmBranch);
    getGitLog(gitLogSettings, function (pLogArray) {
        markdownChangelog += buildChangelogHistoryFromGitArray(pLogArray, maxHistoryLogs, ignoreCommitIds);
        pOnCompleteCallback(markdownChangelog);
    });
}

function buildChangelogHistoryFromGitArray(pLogArray, pMaxHistoryLogs, pExcludeShaArray) {
    var changeHistory = '';
    if (pLogArray.length > 0) {
        changeHistory += '\n## Change History \n\n';
        for (var i = 0; i < pLogArray.length; ++i) {
            var message = pLogArray[i].message;
            if (pExcludeShaArray.indexOf(pLogArray[i].hash) === -1) {
                changeHistory += '* ';
                changeHistory += message;
                changeHistory += '\n';
            }
            if (pMaxHistoryLogs !== -1 && i >= pMaxHistoryLogs) {
                break;
            }
        }
    }
    return changeHistory;
}

function getGitLog(pGitLogSettings, pOnCompleteCallback) {
    git().log(pGitLogSettings, function (err, log) {
        var logArray = null;
        if (!err) {
            logArray = log.all;
        }
        pOnCompleteCallback(logArray);
    });
}

Uploading the build to HockeyApp

To upload the build to HockeyApp, we are using a package called gulp-hockeyapp. It does exactly what we need and you can also decide whether or not you should send an email notification to your HockeyApp distribution groups.

Add this function to your Gulpfile.js that will use all of our previous functions and once everything is done, upload the build to hockeyapp.

function downloadAndUploadToHockeyApp(pBuildTargetId, pHockeyAppID, pOnCompleteCallback, pNotifyAll, pTeams) {
    downloadBuildInformation(pBuildTargetId, function (pBuildInfo) {
        downloadLatestBuild(pBuildInfo, function (pDownloadFilename) {
            createMarkdownReleaseNotes(pBuildInfo, function (pMarkdownChangelog) {
                var options = {
                    id: pHockeyAppID,
                    apiToken: hockeyAppSettings.apiKey,
                    inputFile: paths.dist + pDownloadFilename,
                    notify: pNotifyAll === true ? 2 : 0,
                    status: 2,
                    teamList: pTeams,
                    notes: pMarkdownChangelog,
                    notes_type: 2
                };
                console.log('\nUploading to hockeyapp...\n');
                hockeyApp.upload(options).then(
                    function (response) {
                        console.log("Successfully uploaded build to HockeyApp!");
                        pOnCompleteCallback();
                    },
                    function (err) {
                        throw err;
                    }
                );
            })
        });
    });
}

Creating the Gulp job

By now, we have all the functions that we need. The only thing left is to create a gulp job that uses the function that we just created.

Add the following gulp task to your Gulpfile:

gulp.task(tasks.hockeyApp, function (done) {
    var notifyByEmail = false;
    downloadAndUploadToHockeyApp('android-dev',
        hockeyAppSettings.androidDevKey,
        done,
        notifyByEmail,
        [hockeyAppSettings.devTeamID]);
});

That's it! To run this, just use the following command at the root of your project in the command prompt or terminal to set the job in motion:

D:\YOUR_PROJECT_PATH>: npm run gulp hockeyapp

Ending Notes

As the title suggests, there is much room for improvement with this script. You could split the methods into several files, create unit tests, add better error handling or even use a build computer with jenkins to automatically poll when a new build is successful and run everything automatically.

But at this moment, this is good enough for us. We removed some manual steps and saved everyone in the team some time. We hope that this can be of use to some of you as well!

If you need to have a look at the Gulpfile in it's entirety, you can find it here.

Development Update #31

AURELIAN CLERIC

FIXING BUGS

CAMPAIGN

DIALOGUE SYSTEM & TRADING

BURNED VILLAGE