The 0.7 release used Rake, a Ruby build tool similar to “make”, to manage the build process of Cappuccino applications, as well as Cappuccino and Objective-J itself. In an effort to speed up the tools, reduce the number of dependencies, and allow us to focus on a single language family, we have replaced the Rake dependency with a simple port of Rake for JavaScript, aptly named Jake. Since Jake is written in JavaScript, other JavaScript and Objective-J build tools like the Objective-J compiler can run within the same process, speeding up the build time significantly.

Additionally, we have expanded the command-line/server-side JavaScript environment used for Cappuccino’s tools, now available as a separate project called Narwhal. Narwhal aims to support the emerging CommonJS module and standard library specifications on multiple JavaScript engines.

We started working with the Rhino engine, since the existing build tools were built on Rhino, and have recently added support for the JavaScriptCore / SquirrelFish engine. The performance of Narwhal on JavaScriptCore is an order of magnitude faster than Rhino, vastly improving build times. Currently “narwhal-jsc” supports Mac OS X, but other platform support is in progress.

Another major change was the refactoring of the “press” tool, which attempts to strip unnecessary files from your application bundle. As part of that refactoring we moved the “–flatten” feature, which inlines all code and files into one or more JavaScript files, into a separate tool, unsurprisingly called “flatten”. Flatten now supports splitting your application into multiple files which will be downloaded by the browser in parallel, via the “–split N” option.

For more information on all of these tools, check out the Tools wiki page.

The Problem with Spriting

Image spriting is the act of taking all the images in your app (or framework, or library, or whatever) and combining them down to one single image. This has the advantages of being smaller (since each individual image has overhead associated with the format), as well as allowing you to grab it from your server with one request:

This is faster in an absolute sense, but even more so psychologically since it allows you to show the contents of your app faster without having all the images “flash in” later. Cappuccino currently uses a similar technique with the source code in your application: concatenating it all together and minifying it, but up until now we’ve had no automatic, or built-in, support for spriting, you’ve instead had to do it entirely yourself. And truthfully, no framework has really good support for this.

This is because the fundamental problem with spriting today is that the process isn’t really automated and the results are thus subpar. Sure, there are scripts which will put images together for you, but they all require you to configure them first and to update them as your use of the images in question changes. This is because traditional spriting is dependent on how you use your images. Whether you choose to repeat, stretch, scale, or even draw to a canvas affects which images can be sprited together, and even which can’t be sprited at all. This has a number of unfortunate side effects:

• Rarely do you actually end up with just one image: Instead, you can end up with 2, 3, or even more. This is because images have to be “grouped” by their use. For example, vertically repeating images can be sprited together, but not with horizontally repeating images.
• You may have to actually change your code: Since images are being mutated, the code you write needs to take into account these new images. If you are lucky you have a system that is relatively good at doing this for you. However, if you decide to use an image in a new way (such as drawing it to a canvas), you either have to update your configuration files or choose to code it differently. This is easy to forget.
• Your images are still shipped separately from your code: Even in the best case where you are lucky enough to successfully sprite all your images together, you still have to wait for them separately from when your code is ready, potentially leading to noticeable delays from latency, or a “flash in” effect.
• Inflexible due to loss of data: There exist cases where your code is meant to be used by others, such as with libraries and frameworks. In this case, images can’t be used in any way other than how you intended them to if they are sprited, because the original images are gone or would require a redundant second download.

So unfortunately there is currently no good one-size-fits-all solution for image spriting the way there is with “code spriting”. All of them require the user to actually become involved in the optimization process, and even still can produce less than stellar results. This is clearly not a solution that can scale, and most everyone agrees to this.

But we’re hoping to change this with the release of Cappuccino 0.8, as we’re introducing a whole new, completely cross-platform, way to sprite: base64 images. By encoding images as base64, we create a lossless text representation of images, allowing us not only to use them in whatever way we please, but to actually ship them with the code:

There are many advantages to this:

• One file, guaranteed: All images can always be sprited together regardless of how you plan to use them, and can be included with the actual source code. This has the added benefit that gzip can work its magic on the entirety of your web app as one, producing better results.
• No need to ever modify code or configuration files: Since we’ve eliminated the ambiguous part of spriting images, the Cappuccino build tools are able to perform this optimization on your code automatically without tedious configuration files or having to “learn” how to sprite.

Yes, This Works in IE 6 and 7.

I’m sure most people are wondering how we are pulling this off in versions of IE before 8, since they do not support data URLs. Notice that earlier I didn’t specifically mention data URLs though, I instead only referred to the more broad technology of base64 images. As it turns out, IE has had support for base64 images since version 6 (!) with a little-known technology called MHTML. MHTML allows you place all your resources in one “resources” file, which incidentally can be any file in your website… including the same file that contains all your code.

Cappuccino is already smart enough to be able to automatically download and use different code depending on what browser is being used (and with no server configuration), so we now simply ship data URL versions of this technique to modern browsers, and MHTML versions to older copies of IE:

This is a very exciting feature for us. This has been a weak point in Cappuccino and its nice to finally have a solution that not only works, but is drop dead simple to use. Our tests have been proven incredibly promising, giving us the fastest load times we’ve ever seen with Cappuccino, and absolutely fantastic perceived speed as well. Our tools have all been honed to use this at every level: Apps, frameworks, and themes will automatically sprite your images for you.

This is just one of the many enhancements coming with Cappuccino 0.8, and the best part is as usual you won’t have to change a single line of code to get all the benefits.

Check out the project on GitHub to start hacking on your own real time Cappuccino services!

Atlas is a visual development tool for creating web applications using the Cappuccino framework. The best way to explain Atlas is to show it:

Think Vitamin has an article discussing Atlas in more detail.

We’re really excited about the ways Atlas could change the process of developing web applications. Atlas will allow developers to worry less about boilerplate user interface and glue code, just as the Cappuccino framework gives developers many common features expected by users, letting them focus on the ones specific to their applications.

Furthermore, Atlas allows non-programmers, such as many graphic designers, to join in on the process of actually building an application, rather than just providing mockups that must then be replicated in code by a developer.

Atlas will be available this summer. Sign up for updates on 280atlas.com and we’ll let you know of the progress on Atlas.

In the final installment of the Cappuccino Tool article series (for now), we cover “bake”, an automatic deployment tool. Writing a Cappuccino application does not require using “bake”, but can help with more advanced deployments.

Introduction

“bake” is somewhat analogous to Capistrano, a deployment tool often used for deploying Ruby on Rails and other applications, but “bake” has several features specifically for deploying Cappuccino applications.

The basic idea behind “bake” is to assemble a complete deployable copy of your application by pulling the pieces from various places (Git, Subversion, and local or remote directories via rsync), building it, packing it up, and deploying it to your server(s).

One key feature of bake is the management of multiple deployments over time, which allows three things: atomic deployments, keeping your client side code (the Cappuccino application) synchronized with your server side code, and enabling aggressive caching on all your static resources (code, images, etc). We’ll see later this is done by putting all the new resources in place, then “swinging” a single file, the index.html with a <base> tag.

“Atomic deployments” means that the deployment is updated to the new version in a single operation, which is either successful or not, nothing in between. Every client loading your application receives either the previous deployment, or the new deployment, but never a mixture of the two.

Keeping client code synchronized with server code is (sometimes) important for making sure users who have already loaded the application can continue using it even after you deploy a new version of the server side code that is incompatible with older versions of the client side code.

Aggressive caching of static resources reduces the load time of Cappuccino applications for repeat useres. This is possible because every resources URL is unique in each deployment, so we can set the “Expires” header far in the future (e.x. ‘Expires “Thu, 15 Apr 2010 20:00:00 GMT”‘). While the actual file name is not unique (for example Objective-J/Objective-J.js), in the deployed application the path will be unique (www.yourserver.com/YourApp/1225273279/Objective-J/Objective-J.js where the version number changes for each deployment)

Summary

The configuration for a bake deployment is specified in a “bakefile”, which is in the JSON format with a structure like the following:

{
	"sources" : [
	    {
	        "type"    : "git",
	        "path"    : "git://github.com/280north/cappuccino.git",
	        "parts"   : [
    	        {
    	            "src"       :   "Objective-J",
    	            "dst"       :   "Frameworks/Objective-J",
    	            "build"     :   "ant -DBuild=BUILD_PATH",
    	            "copyFrom"  :   "Release/Objective-J"
    	        },
	            {
    	            "src"       :   "Foundation",
    	            "dst"       :   "Frameworks/Foundation",
    	            "build"     :   "steam build -c Release -b BUILD_PATH",
    	            "copyFrom"  :   "Release/Foundation"
	            },
        	    {
    	            "src"       :   "AppKit",
    	            "dst"       :   "Frameworks/AppKit",
    	            "build"     :   "steam build -c Release -b BUILD_PATH",
    	            "copyFrom"  :   "Release/AppKit"
        	    }
	        ]
	    },
	    {
	        "type"    : "rsync",
	        "path"    : "/Users/username/projects/NewApplication",
	        "parts"   : [
	            {
    	            "src"       :   "",
    	            "dst"       :   "Blah"
	            }
	        ]
	    },
	    {
	        "type"    : "svn",
	        "path"    : "https://svn.youserver.com/Project/trunk",
	        "parts"   : [
	            {
    	            "src"       :   "subdirectory",
    	            "dst"       :   "Something"
	            }
	        ]
	    }
	],
	"deployments" : [
	    { "host" : "deploy@myserver.com", "path" : "/var/www/mysite/public" }
	],
	"templateVars" : {
        "APPLICATION_NAME" : "My Application",
        "BACKGROUND_COLOR" : "black",
        "TEXT_COLOR" : "black"
	}
}

This bakefile contains three “sources”: a git repository, a local directory, and a svn repository. The git repository is the main Cappuccino git repository, hosted on GitHub. You could replace this with a different branch if you needed a specific version, or even your own if you have one. This source contains three “parts”, Objective-J, AppKit, and Foundation. The “src” specifies where in the checkout the part is located, while “dst” specified where in the final built application it should be placed. You can optionally specify a “build” command, which is run from the specified “src” directory. This command should include the placeholder “BUILD_PATH”, which will be filled in by “bake” with the temporary directory where the build results are placed (equivalent to $STEAM_BUILD for any “steam” commands). Additionally, if you specify a “build” command you also need to specify a “copyFrom” parameter, which tells build the subdirectory of the build directory it should copy the built part from.

Once each piece of the application has been built it is copied to the appropriate subdirectory of a uniquely named (using the current Unix timestamp) “versioned” directory. Rather than deploying this versioned directory directly, it is placed within another directory which contains a special index.html file. This file includes a <base> tag that points to the versioned directory, which causes the browser to treat it as the base for all relative URLs, thus loading the correct version of all resources without requiring the application be located at a unique URL. “bake” will fill in “$VERSION” template variable in the base tag of the template with the correct

Additionally, “bake” will calculate the size of all code which needs to be loaded, in order to provide an accurate loading progress bar. The calculated size is inserted into the template variable “$FILES_TOTAL”. You can also specify other template variables such as “$APPLICATION_NAME”, “$BACKGROUND_COLOR”, and “$TEXT_COLOR” in the bakefile.

If you use your own template, the only required variable is the “$VERSION” one, but you can specify as many custom variables as you wish.

Finally, once the application has been built and assembled, it is optionally run through “press”, then gzipped. If the “–deploy” command line parameter was given then the gzipped application is transmitted to the deployment servers, un-gzipped, and copied to the deployment path using a specific sequence of command to ensure the deployment is atomic:

tar xzf 1225273279.tar.gz
mkdir -p /path/to/deployment
mv 1225273279/1225273279/ /path/to/deployment/1225273279
mv 1225273279/index.html path/to/deployment/index.html
rm "1225273279.tar.gz
rmdir 1225273279
cd /path/to/deployment
ln -nsf 1225273279 Current

(version “1225273279″, deplyoment directory “/path/to/deployment”)

One important thing to note about “bake” is that since the old deployment remains live (but hidden) if you deploy a fix for a security vulnerability you should manually purge all the old deployments from your server. If the change doesn’t introduce any incompatibilities between the server and client, you could set up a rewrite rule to direct all requests to the purged deployments to the latest, which is always symlinked to “Current”.

Conclusion

Once again I should stress that this tool is by no means required for writing a Cappuccino application. It was written to help deploy our production application, 280 Slides, but we have made it available for anyone who has similar deployment requirements. If it doesn’t quite fit your requirements, please feel free to suggest improvements, or even better, make improvements yourself and contribute them back!

In the third installment of the build tools series we tackle one of the most advanced build tools: “press”. Recall from the overview that “press” is a tool for stripping code and other optimizations. Additionally, it can convert an entire Objective-J application into a single pure JavaScript file, which we’ll see can be useful.

press

The primary goal of “press” is to strip unused code from your application and frameworks and perform other optimizations. It needs to know about the entire application, including your application code and the Cappuccino frameworks, in order to determine the dependencies and remove unused files. It is called with two parameters, the path to your application, and the output directory:

press root_directory output_directory

This will run “press” on your application located at root_directory, and output a similar but smaller version (with unused files stripped) at output_directory.

Additionally, there are several useful options:

• ‐‐main path: The relative path (from root_directory) to the main file (default: ‘main.j’)
• ‐‐frameworks path: The relative path (from root_directory) to the frameworks directory (default: ‘Frameworks’)
• ‐‐png: Run pngcrush on all PNGs (pngcrush must be installed!)
• ‐‐flatten: Flatten all code into a single Application.js file and attempt to add script tag to index.html (useful for Adobe AIR and CDN deployment)
• ‐‐nostrip: Don’t strip any files. Mostly useful for debugging.

‐‐flatten

The most interesting of these options is “‐‐flatten”, which converts your application into a single pure JavaScript file that can be imported with a standard “script” tag. While the Objective-J load system is great in most scenarios, there are currently a few limitations that a pure JavaScript file helps overcome. Namely that the load system doesn’t work in Adobe AIR due to strict security restrictions, and the load system doesn’t work across domains, such as with a CDN, also due to browser security restrictions. By flattening all Objective-J code into a JavaScript file, we can use a standard script tag to load our application, which works cross-domain and in Adobe AIR.

Internals

If you’re interested in how “press” works, read on. It starts by loading your application while noting which global variables are defined in each file. It then “walks” along the dependency “graph”, including every imported file, with one important exception: framework imports like “import <AppKit/AppKit.j>” and “import <Foundation/Foundation.j>” are ignored since these types of imports would result in the entire framework (including unused files) being imported. Instead, press looks at the global dependencies, and only keeps the files that are referenced by any other file that has been imported.

Conclusion

“press” is a already a great tool, but there is much room for improvement. In the future we would like to add finer-granularity code stripping, such as method-level rather than file-level. Additionally, we’re working on an automatic image spriting and optimization feature which determines exactly which images are required for your application and includes them in a single image, vastly reducing the number of HTTP requests to load your application.

In our previous post on the Cappuccino build tools we summarized the purpose of four of the tools. In this article we’ll dive deeper into two of them, objjc and steam.

To review, the purpose of these tools is to prepare your application for deployment by pre-compiling all application Objective-J code into JavaScript, thereby reducing the load time for clients. Additionally, since most developers won’t need to modify Cappuccino itself, we always compile Cappuccino’s AppKit and Foundation frameworks to further reduce load time.

The only time you’ll need to run steam (which handles running objjc for you) is when you’re deploying your application, or if you’re making changes to the Cappuccino frameworks themselves. In the latter case, the easiest way to recompile the frameworks is to run the default ant task in the Cappuccino source tree:

cd /path/to/cappuccino
ant

To use the freshly built frameworks simply replace the “Frameworks” directory in your application with a symbolic link to the Release directory in the directory pointed to by your $STEAM_BUILD environment variable:

cd /path/to/your_application
mv Frameworks Frameworks-Original
ln -s $STEAM_BUILD/Release Frameworks

(if you’re loading your project through a webserver like Apache make sure to enable the FollowSymLinks option)

objjc

objjc, the Objective-J compiler, is the most important of the build tools, but you’ll likely never invoke it directly. objj can take any number of parameters as input filenames and output filenames (preceded by the “-o” flag). For example:

objj Class1.j -o Class1.o Class2.j -o Class2.o

This will invoke the Objective-J compiler (which is identical to the one used by Objective-J in the browser) on each input file, outputting JavaScript plus some metadata to each specified output file. Next we need to combine all of these individual output files into a single “.sj” archive that contains the JavaScript and import dependency metadata for the entire framework. It would be possible to do this by hand, but steam takes care of it for you.

steam

steam is the general Cappuccino build tool that manages the creation and compilation of your Cappuccino applications.

To create a basic Cappuccino application, simply run steam with the “create” command:

steam create ApplicationName

This will create a new Cappuccino application in the directory “ApplicationName”. This application can be modified and run (by opening index.html in the browser) without any further usage of the build tools.

When you’re ready to compile the application code for deployment, add a .steam file to your application, like the following:

This file defines the targets and configurations for your project. This one defines a single target, YourApplication, and two configurations, Release and Debug. To compile your application, simply execute steam with the build command, the .steam filename, and the configuration name:

steam build -f MyApp.steam -c Release

If a .steam filename is not provided it will look for a file with the extension “.steam”. Likewise, if a configuration name is not provided, it will use the first one defined in the file. The following will run the “Debug” configuration in your .steam file:

steam build

steam gathers all “.j” files in your application, compiles them, and combines them into a single “.sj” file (“static Objective-J”), along with a new “Info.plist” which tell Objective-J which files are contained in the “.sj”.

The results are placed in a subdirectory of $STEAM_BUILD named the same as the configuration, i.e. Debug or Release. Alternatively, pass the “-b” flag and a build directory to specify where it should be built.

The compiled application or framework can then be used in place of the original uncompiled collection of .j files. If it was an application, copy or symlink your Frameworks directory to it’s directory. If it was a framework, copy it to your application’s Frameworks directory.

Conclusion

This article covered the core Cappuccino build tools, objjc and steam. Remember that you will probably never want to call objjc directly, but rather rely on the “steam build” command to manage the build process for you. Additionally, remember the build process is entirely optional (except if you’re editing the Cappuccino frameworks themselves), and is only necessary as a deploy-time optimization.

In subsequent articles we’ll cover the remaining more advanced build tools, press and bake, as well as further deployment optimizations.

This article is a high level overview of the current Cappuccino and Objective-J build tools. Subsequent posts will cover each tool in more detail.

One of our primary design goals with Cappuccino and Objective-J was to keep the simple development cycle web developers are used to: edit your code, save, refresh the browser. At the same time we wanted to add powerful language features to JavaScript without requiring the user install a plugin like Flash or Silverlight. At first glance it seemed like only two of these three requirements could be satisfied simultaneously, until we had the key realization that we could simply write our compiler in JavaScript itself, and perform the compilation at runtime on the client.

In reality the Objective-J “compiler” is more like a preprocessor, performing a relatively simple transformation from Objective-J code to JavaScript code, which is then interpretted by the browser’s native JavaScript engine. We don’t do full parsing and compiling, nor do we have a separate interpreter on top of JavaScript. The result is that the compiler and resulting code run very fast.

This turned out to work great. Simply download the starter package, load index.html in your browser, and start editing away! If you’re just getting started with Objective-J and Cappuccino you don’t even need to worry about the build tools, they’re entirely optional.

When it comes time to deploy your application you may want to optimize it. There’s no reason for your customers to wait any longer than necessary, even if the overhead is minimal. That’s where the build tools come in. Each tool has a specific purpose, and they all work together to produce an optimized Cappuccino application:

• objjc – the Objective-J compiler
• steam – a general tool for managing the build process, creating new Cappuccino applications, etc.
• press – code stripping and other optimizations
• bake – deployment tool

objjc

At the lowest level is “objjc”, the Objective-J compiler. It’s equivalent to gcc or javac, except it converts Objective-J code to JavaScript. Because we can run the same JavaScript code in the browser and Rhino the core of objjc is identical to the in-browser compiler.

Currently objjc’s implementation is closer to a simple preprocessor than a compiler, much like the original Objective-C compiler was implemented as a preprocessor on top of C. The preprocessor simply looks for the few Objective-J syntax additions and replaces it with the raw JavaScript equivalent. A big reason this is possible is that Objective-J is a strict superset of JavaScript, thus all JavaScript code is valid Objective-J code.

Typically you don’t call objjc directly, but rather let steam manage the build process.

steam

Next up is steam, which is the main Objective-J and Cappuccino build tool. Currently it serves two purposes: “steam build” (similar to Xcode’s “xcodebuild”) and “steam create” (similar to Ruby on Rail’s “rails” command).

“steam build” runs the compilation process on an entire application or framework and outputs a single “.sj” file, vastly decreasing the number of HTTP requests required to load an application. These “.sj” files are archives of the compiled input “.j” files, along with information detailing each file’s required imports. This allows the Objective-J loading system to efficiently and asynchronously load a large application.

“steam create” copies all the files required for a new project to the directory specified.

press

The “press” tool takes in a full application and attempts to determine which code files are unecessary. It then strips those files and writes the results to another directory. This could be thought of as a “linker” though it works very differently than a traditional linker. On a simple project it will reduce the AppKit and Foundation frameworks size by approximately 30%, and we expect further improvements.

Optionally, it can run the pngcrush utility on all your png graphics, attempting to reduce their size. Eventually press will also automatically sprite images, further reducing the number of HTTP requests required to load your images.

bake

Finally, we have “bake”, the Cappuccino deployment tool (like “Capistrano” but specialized for Cappuccino applications). Bake orchestrates all the above tools to produce, and optionally deploy, a highly optimized Cappuccino application. First it can pull both Cappuccino and your application code from git, svn, or a local or remote directory via rsync.

It then copies pieces of the various “checkouts” to the “products” directory, first running a build process like “steam” or ant, if specified. For example, it can run steam on your application code, and copy the results to the product directory, then do the same for AppKit and Foundation, copying the results to the Frameworks/AppKit and Frameworks/Foundation directories to build the complete application.

Once an application is assembled, bake can run “press”, optimizing the application’s size.

Then bake sets up a directory structure and index.html file which allows for several important tricks: aggressive caching, keeping the client application in sync with the server backend, and atomic deployments, all while preserving a single user-facing URL for the application.

Finally, the application is archived and gzipped, scp’d to one or more servers, and atomically deployed to the specified path(s).

Conclusion

Cappuccino’s build tools are great at optimizing your deployed application, but there are further web server specific optimizations possible as well, including enabling gzipping and caching. These will be discussed in subsequent articles.