Popular system build tools

From Knowledge Kitchen
Jump to: navigation, search


The process of building a software system typically involves running several separate software tools to perform tasks like compiling all the components that together define the system, linking everything up, setting up the environment within which the system will be run, preparing any data required by the system to function properly, and keeping track of the dependencies that one component has on another, etc. Anything that is necessary for the software to be executable in its intended form is considered building.

Build tools

In a production environment, these steps are usually managed by a build tool. Build tools automate the process of building by running scripts that coordinate all these necessary programs, running them in the proper sequence necessary to function correctly, supplying the necessary inputs to each tool, and ensuring that each succeeds at performing its intended function before running the next.

The build utility typically needs to compile the various files, in the correct order. If the source code in a particular file has not changed then it may not need to be recompiled (may not rather than need not because it may itself depend on other files that have changed). Sophisticated build utilities and linkers attempt to refrain from recompiling code that does not need it, to shorten the time required to complete the build. A more complex process may involve other programs producing code or data as part of the build process.

This document gives a high-level overview of some popular build tools in use today. As you'll see there is an evolutionary history of build tools, where each generation tries to solve the limitations of the previous.

Make

Stuart Feldman, creator of Make

Make, and the many automation tools derived from the original make, is considered a classic build automation tool. Make's widespread usage is, in part, due to its inclusion in UNIX since 1976. Make was originally hacked together in a weekend by an engineer at Bell Labs, where UNIX was created, in order to solve the all-too-common problem of developers wasting time debugging builds where the problem was human error in performing the build, not software bugs in the code.

Make's major value propostion was/is:

  • Make executes a single build script that describes all the steps needed to build a system
  • Make provides a mechanism for keeping track of depencies among components, which until that time had been handled manually.
  • Make also speeds up builds by avoiding redundant build steps. It does this by detecting the modification date of the components it has instructions to build, comparing those modification dates to the date of the last build, and only rebuilding those components that have been modified since the last build.

Up until Make's introduction, builds were performed step-by-step manually in an ad-hoc manner with no standardized way to keep track of dependencies, leading many developers to accidentally skip steps but have no record of what steps they had skipped.

Key features:

  • written for UNIX
  • tracks dependencies
  • intelligently skips build steps that are redundant
  • most commonly used for C and C++, but supports virtually any language
  • can be used for more than just builds - automates any arbitrary sequence of shell commands
  • build scripts written as Makefiles
  • many spin-offs, such as GNU make, Cmake, and more

Makefiles

Make's build automation scripts are termed Makefiles. As Stuart Feldman, the creator of Make, has stated:

"Makefiles were text files, not magically encoded binaries, because that was the Unix ethos: printable, debuggable, understandable stuff."
- Stuart Feldman, creator of Make

The core of a Makefile consists of rules. Each rule defines a target to be built, including a list of all the components (files or other targets) upon which that target depends.

  • for example, a compiled Java byte code target (e.g. *.class) depends upon a Java source code file (e.g. *.java). So a rule that defines a byte code file target would include the java source code file upon which it depends.
  • a system that is composed of several Java byte code files would include a rule that includes all the other byte code file targets as its dependencies. This would require each individual byte code file target to be built in order for the entire system to be considered built.

Each rule may include, below it, an indented list of commands that are to be run in order to convert the dependencies (usually source code files) into the target (usually the executable).

Executing make

Makefile scripts are executed from the command-line with the 'make' command.

  • 'make' will invoke the first rule in the Makefile
  • 'make rulename' will specifically invoke the rule named 'rulename'

Example

The following Makefile example includes instructions for how to compile three Java byte code class files, including:

  • definitions - some variables that contain useful definitions of commands and options to be run
  • rules - instructions for how to perform the builds of each target
  • cleanup - removing artifacts of the build so the next build starts from a clean slate
# a variable that defines the java compiler command to use
JCC = javac

# a variable that refers to compilation flags to be sent to the compiler
# the -g flag compiles with debugging information
JFLAGS = -g

# typing 'make' with no options from the command-line will, by default, invoke the first rule in the Makefile, here called 'default'.
# this rule defines the target, 'default', and the other targets upon which it depends
# this rule does not include any commands necessary to build the default target, only the other targets upon which this target depends in order to be complete
default: Average.class Convert.class Volume.class

# this rules defines the target named 'Average.class', which you'll notice is one of the dependencies of the 'default' target
# the Average.class target is dependent on the Average.java source code file
# this rule includes indendented commands below it, which outline the command-line operations to be performed to build this target
# by substituting the variables with their values, this command would look like 'javac -g Average.java'
Average.class: Average.java
        $(JCC) $(JFLAGS) Average.java

# this rule follows the same pattern as the previous rule
Convert.class: Convert.java
        $(JCC) $(JFLAGS) Convert.java

# this rule follows the same pattern as the previous rule
Volume.class: Volume.java
        $(JCC) $(JFLAGS) Volume.java

# the following rule is not invoked in a default build, since it is not required by the default rule
# in order to invoke the following rule, the command 'make clean' must be run from the command-line.
# this rule removes all .class files, so that the next make rebuilds them from scratch
# note that our code has not defined the RM variable, since Make already includes it in its pre-defined variable set, although you could define the variable manually if desired
clean: 
        $(RM) *.class

Here is an alternate example Makefile that performs the same actions as the previous example, but using more advanced syntax features of Makefiles to remove redundancy:

JCC = javac
JFLAGS = -g

# Make includes a built-in set of common compiled targets and their source code file dependencies (e.g. .o files depend upon .c files in C, or .class files depend upon .java files for Java)
# This command clears any such defaults so we can explicitly define how to build .class files from .java files
.SUFFIXES: .java .class

# a macro defining how to build .class targets from .java files.
# $* is a macro that is replaced with the basename of whichever target is being built from this macro
.java.class:
        $(JCC) $(JFLAGS) $*.java

# a macro containing 4 items (one for each source code file to be built)
CLASSES = \
        Average.java \
        Convert.java \
        Volume.java

# the default target that depends upon the classes target
default: classes

# the target that 'does the work' in this file.
# it includes a macro that takes each of the items from the CLASSES macro and replaces its .java extension with a .classes extension using the .java.class macro defined above.
classes: $(CLASSES:.java=.class)

clean:
        $(RM) *.class

Limitations

"If you move stuff around and do more than just invoking the compiler (and even then), the platform-specific stuff gets very awkward to handle in make. And having a makefile which only works on one system isn't very nice for a cross-platform language"
- Joey, from Stack Overflow

There are several common criticisms of Make

  • Makefile instructions are written as shell commands, which are inherently platform-specific. So, for example, a Makefile for Windows will require separate commands from a Makefile for UNIX. Or a Makefile written for a bash shell will be different from one written for the alternative tcsh shell.
  • Make does not deal well with files spread across a complex directory structure, such as often the case in Java and many other languages and platforms.

Partially to address this and other limitations, GNU also releases Automake, a tool to automate the process of writing the Makefile automation scripts. Automake allows developers to write Makefiles in a higher-level language, where some depenencies can be automatically detected by analyzing the source code and Makefiles appropriate for the target platform are generated.

  • Despite this, many developers in languages besides C or C++ often prefer other more flexible build tools

Ant

Apache Ant, begun in 2000, is a popular automated build tool that was designed explicitly to address some of the shortcomings of Make for Java builds.

Key features:

  • open source
  • similar to Make, but designed for Java projects
  • built in Java, so requires the Java Development Kit is installed
  • makes it easy to integrate unit testing using Java's popular JUnit testing framework.
  • build scripts written as build.xml files

Whereas Make build scripts are written in obscure Makefile syntax, Ant build scripts are specified in XML, a more general-purpose syntax that is understood (although not necessarily liked) by most developers today.

  • Ant build scripts are saved in a file named build.xml

Although some of Ant's XML tag names, like 'mkdir', seem to imply UNIX-specific actions, the specific shell commands used to implement these XML instructions are not dictated by the XML code.

  • Ant executes the appropriate shell commands to achieve the stated goal, given the current platform on which it is running.

Example

The following build.xml example contains similar instructions to the Makefile examples above, but using Ant's XML syntax. This example includes four different targets:

  • compile - creates a 'classes' directory if necessary, and compiles the Java source code files into Java byte code placed in that directory
  • clean - removes the 'classes' directory, thereby wiping out any previously compiled Java byte code files
  • jar - packages up the project into a Java Archive (.jar) file for distribution
  • clobber - deletes any previously-created Java Archive (.jar file)
<?xml version="1.0"?>
<project name="Calculator" default="compile">
    <target name="compile" description="compile the Java source code to class files">
        <mkdir dir="classes"/>
        <javac srcdir="." destdir="classes"/>
    </target>
    <target name="clean" description="remove previously built files">
        <delete dir="classes"/>
    </target>
    <target name="jar" depends="compile" description="create a Jar file for the application">
        <jar destfile="hello.jar">
            <fileset dir="classes" includes="**/*.class"/>
            <manifest>
                <attribute name="Main-Class" value="CalculatorProgram"/>
            </manifest>
        </jar>
    </target>
    <target name="clobber" depends="clean" description="remove all artifact files">
        <delete file="hello.jar"/>
    </target>
</project>

Limitations

While Ant was built to overcome limitations in Make, it receives its own set of criticisms from developer communities.

  • Despite being flexible and extensible, XML today is generally considered to be an overly verbose markup language with a lot of redundancy. Some XML build scripts can seem unnecessarily complex.
  • Each build script tends to be custom and dissimilar from any other... there is little convention in scripts
  • Ant suffers from legacy Java compatibility issues: some tasks that Ant supports, such as javac and java, use default argument options that are no longer current with the latest Java. Changing the implementation of these would break older code, so the developers have not updated Ant to more modern usage conventions.

Maven

Apache Maven is a successor to Apache Ant, initially released in 2004, that aims to address some of Ant's shortcomings, plus adding some additional features. Like Ant, Maven scripts are written in XML.

The fundamental difference between Maven and Ant is that Maven assumes all projects will be set up following a conventional structure and their build will follow a conventional workflow for tasks like getting resources from source control, compiling the project using conventional tools, unit testing the usual way, and so on.

Key features:

  • open source
  • convention over configuration - Maven's build scripts follow pre-defined conventions relevant to the type of project being built. For example, build script writers for a Java project need not specify the commands to compile Java source code files to byte code files... Maven will know to do this already, given the type of project.
  • plug-in architecture - Maven is created with a plug-in architecture that allows 3rd party plug-in developers to add integrate with any tool that can be executed from the command-line.
  • language agnostic - support for several languages is included by default, and developers can build plug-ins to add Maven support for whatever additional language, platform, or tool they desire to work with
    • in actuality, Maven is still primarily used for Java projects
  • Apache hosts a repository of such plug-ins that are easily installed with a built-in package manager
  • build scripts written as Project Object Model files written in XML, named pom.xml

Maven build lifecycle

Maven breaks down each build into multiple phases. A pom.xml file can specify what tasks (called 'goals' in Maven terminology) to perform in each phase, or it can just rely on the default goals Maven understands as conventioanl for any phase. The default lifecycle phases are:

  • validate - validate the project is correct and all necessary information is available
  • compile - compile the source code of the project
  • test - test the compiled source code using a suitable unit testing framework. These tests should not require the code be packaged or deployed
  • package - take the compiled code and package it in its distributable format, such as a JAR.
  • integration-test - process and deploy the package if necessary into an environment where integration tests can be run
  • verify - run any checks to verify the package is valid and meets quality criteria
  • install - install the package into the local repository, for use as a dependency in other projects locally
  • deploy - done in an integration or release environment, copies the final package to the remote repository for sharing with other developers and projects.

Each build goal specified in a Maven pom.xml file can be marked as being relevant to a particular phase

  • Maven automatically executes these phases in order
  • For example, any goals marked as being relevant to the 'compile' phase will always executed before any goals marked as for the 'test' phase, which will always be executed before the 'package' phase goals, etc.
  • If a Maven pom.xml file only includes a goal related to the 'package' phase, and none others, Maven will automatically execute all previous phases in the Maven Lifecycle, such as 'compile' and 'test' in the conventional way prior to fulfilling the stated 'package' phase goal.

Examples

The following pom.xml file shows the reduced syntax that Maven requires, as compared to the similar build functionality in the Ant example above. Maven works in a conventional way, unless told to do otherwise. So basic instructions on how to find source code and compile it, for example, are not required if one desires to stick to a conventional workflow.

<project xmlns="http://maven.apache.org/POM/4.0.0"   
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"  
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0   
http://maven.apache.org/xsd/maven-4.0.0.xsd">  
  
  <modelVersion>4.0.0</modelVersion>  
  <groupId>edu.nyu.cs.fb1258</groupId>  
  <artifactId>calculator-app</artifactId>  
  <version>1</version>  
  
</project>

While the above shows the minimum build script for a simple project, Kevin Sawkicki's GitHub/Maven example pom.xml file is a good example of a more 'customized' Project Object Model XML file (pom.xml) that includes information about the project's authors, copyright license, workflow, and loads plugins to automatically pull resources from a GitHub repository and build the contents.

Limitations

As with any tool, Maven has its critics. Common complaints:

  • Maven requires a rigid adherence to what it considers conventional. Therefore, unusually organized or complex unconventional projects may require a lot work overriding the defaults
  • Because Maven is built to follow a conventional build with no need for explicit instructions on how to do that, it is not easy for developers to know what it is doing

Gradle

Gradle is [yet another] open source build automation tool primarily focused on Java projects. It has many similarities to Maven and Ant before it:

  • Gradle, like Maven, emphasizes convention over configuration
  • Gradle, more so than Maven, allows developers to override convention when needed.
  • Gradle, like Maven, is built with a plug-in architecture that allows 3rd-party developers to build integration and support for other languages, platforms, and tools besides standard Java
  • Gradle, like Maven, hosts plug-ins and common project dependencies in hosted repositories that all projects can pull from
  • Gradle, like Maven and to some degree Ant, supports Agile development trends, such as continuous integration and continuous deployment
  • Gradle, like Maven and Ant, is open source

Each Gradle language plugin has a built-in set of tasks it can perform for projects in that language without requiring additional instruction. This is handled by following standard conventions for that language, much like Maven's goals do. The following are the most significant tasks included in Gradle's Java plugin:

  • assemble - Assembles the outputs of this project (this includes packaging it into a .jar file).
  • build - Assembles and tests this project.
  • clean - Deletes the build directory.
  • jar - Assembles a jar archive containing the main classes.

Gradle has three key differences from Maven and Ant:

  • custom logic in simple build scripts, written in either Java or Groovy (a syntactically similar language to Python or Ruby that is compiled to Java byte code and therefore interoperable with Java code), as opposed to verbose XML with no decision structures as in Ant and Maven
  • support for integrating legacy build scripts, such as those from Ant and Maven, into Gradle projects
  • facilitates complicated build scripts involving a diverse technology stack, such as a build for a client-side component in Javascript speaking with a Java back-end that in turn sends calls to an old legacy C++ application
  • build scripts saved in a file named build.gradle

Gradle for Python

In 2016, LinkedIn released {py}gradle, a variant of Gradle they had created to be used specifically for Python projects.

Besides pygradle, the Python ecosystem lacks an automated build tool equivalent to Java's Maven or Gradle.

  • there are a variety Python build tools, such as Waf, Scons, and Distutils. But Python's build tool ecosystem has tended to lag behind that of Java in terms of its sophistication.

Example

The following build.gradle file informs Gradle to use the Java plugin (including its set of default Java tasks) and informs Gradle which class file is the main one. This is all that is necessary to build a standard Java program.

apply plugin: 'java'

jar {
    manifest {
        attributes 'Main-Class': 'edu.nyu.cs.fb1258.CalculatorProgram'
    }
}

To build this program, one need only execute the following command:

gradle assemble

To build and test this program:

gradle build

These commands create a .jar file named after the directory in which the project is saved.

To execute the .jar file that the build produces:

java -jar calculator-program.jar

Grunt

Grunt is a Javascript-specific build automation tool, started in 2012, that can be setup to automate repetitive build-related tasks like minification of code, compilation, unit testing, code linting, etc.

Grunt takes the concepts of other automation tools and brings them into the Javascript ecosystem.

Key features:

  • Like the other build automation systems, developers can configure tasks that a Grunt script can perform and define the operations to be included in each task.
  • Grunt is built on top of Node.js, the popular Javascript framework.
  • Grunt supports a plug-in architecture that allows seamless integration with other Javascript-based development tools like unit testing, code analysis, etc.
  • Continuing the path tread by Gradle, Grunt also abandons XML as a build script language. In its place, Grunt uses Javascript, which will be familiar to Node.js programmers.
  • Grunt automation scripts are written into a Gruntfile.js file.

Example

The following Gruntfile.js shows a standard setup of automation for a web application. This build script loads and configures each of the following Grunt plug-ins:

  • grunt-contrib-uglify - used to minify the code so that extra whitespaces are removed and names of things are reduced to bare minimum
  • grunt-contrib-qunit - used to do unit testing of the code
  • grunt-contrib-concat - used to merge multiple javascript files into one
  • grunt-contrib-jshint - used to do static code analysis
  • grunt-contrib-watch - used to watch every commit of new code and trigger the tools above to act on it
module.exports = function(grunt) {

  grunt.initConfig({
    pkg: grunt.file.readJSON('package.json'),
    concat: {
      options: {
        separator: ';'
      },
      dist: {
        src: ['src/**/*.js'],
        dest: 'dist/<%= pkg.name %>.js'
      }
    },
    uglify: {
      options: {
        banner: '/*! <%= pkg.name %> <%= grunt.template.today("dd-mm-yyyy") %> */\n'
      },
      dist: {
        files: {
          'dist/<%= pkg.name %>.min.js': ['<%= concat.dist.dest %>']
        }
      }
    },
    qunit: {
      files: ['test/**/*.html']
    },
    jshint: {
      files: ['Gruntfile.js', 'src/**/*.js', 'test/**/*.js'],
      options: {
        // options here to override JSHint defaults
        globals: {
          jQuery: true,
          console: true,
          module: true,
          document: true
        }
      }
    },
    watch: {
      files: ['<%= jshint.files %>'],
      tasks: ['jshint', 'qunit']
    }
  });

  grunt.loadNpmTasks('grunt-contrib-uglify');
  grunt.loadNpmTasks('grunt-contrib-jshint');
  grunt.loadNpmTasks('grunt-contrib-qunit');
  grunt.loadNpmTasks('grunt-contrib-watch');
  grunt.loadNpmTasks('grunt-contrib-concat');

  grunt.registerTask('test', ['jshint', 'qunit']);

  grunt.registerTask('default', ['jshint', 'qunit', 'concat', 'uglify']);

};

References

What links here