knowledge-kitchen

System Build Tools - Make, Maven, Gradle, NPM, Pip, Grunt, Gulp, etc.

System building in preparation for execution.

1. Overview
2. Common Features of Build Tools
3. Stages
4. Continuous Integration
5. Make: The Original Build Tool
6. Java Build Tools
7. … Intermission …
8. JavaScript Build Tools
9. Python Build Tools
10. Conclusions

Overview

Concept

System building is the process of creating complete, executable system by compiling and linking the system components, external libraries, configuration files, and other information.

Challenge

System building must take into account many complex environmental and programmatic factors, and steps must often be performed in a particular sequence in order for the build to succeed.

Solution

Automated build tools are generally necessary to handle the complexity.

Common Features of Build Tools

Build Script Generation

The build system should analyze the program that is being built, identify dependent components, and automatically generate a build script (configuration file).

The system should also support the manual creation and editing of these build scripts.

Version Control Integration

The build system should be able to check out the required versions of components from a version control system.

Minimal Recompilation

The build system should work out what source code needs to be recompiled and set up compilations as necessary.

Executable System Creation

The build system should link the compiled object code files with each other and with other required files, such as libraries and configuration files, to create an executable system

Test Automation

Build systems should be able to integrate with test automation tools such as xUnit other similar unit testing tools which check that the build has not been “broken” by the changes.

Reporting

The build system should provide reports about the success or failure of the build and the tests that have been run.

Documentation

The build system configuration script serves as a form of documentation

• At the very least, a build configuration script documents the dependencies of the system

• It also documents the order in which all steps of the build must occur

• A build system may also be able to generate release notes about the build and system help pages automatically.

Stages

Developer’s Machine

Developers may wish to test and build locally before performing the ‘official’ build.

They will typically check out the latest version from the version control system and try to build it locally.

The developer’s machine often differs significantly from the target machine that code will be deployed to.

Build Server

Used to build the definitive version of the system.

Should integrate with the version control system to build the latest.

Target Machine

The platform on which the system is ultimately intended to execute. This may be significantly different from the development machines and build servers.

Keeping the developer’s machines and the build server environment in close emulation of the Target Environment is a whole ‘nother area of automation.

In cases where the target machines differ significantly from the others, testing cannot be complete until done on a target machine.

Continuous Integration

Automated Build and Test

Agile methodology promotes the practice of very frequent automated builds and automated testing once the build is done.

• Such automated or semi-automated processes in the software engineering lifecycle are collectively known as continuous integration (CI).

• GitHub Actions, Jenkins, and Circle CI are a few examples of popular Continuous Integration tools - there are many others as well.

The Original Build Tool

Make

Make, included in UNIX starting in 1976, and the many automation tools derived from the original make, is considered a classic build automation tool.

• Created by Stuart Feldman to solve the all-too-common problem of developers wasting time debugging builds where the problem was human error in performing the build, not software bugs in the code.

Make (continued)

Key features:

• used primarily for C and C++ development, but can be used as generaal purpose automation tool

• build script written in an idiosyncratic script file named Makefile

• allows multiple build targets

• tracks dependencies

• performs minimal recompilation

Make (continued again)

See…

• an example of a simple Makefile.
• a slightly more advanced example that achieves the same build but removes some redundancy.

Make (continued once more)

“If you move stuff around and do more than just invoking the compiler (and even then), the platform-specific stuff gets very awkward to handle in make. And having a makefile which only works on one system isn’t very nice for a cross-platform language”

-Joey, from Stack Overflow

Make has a few limitations that have led to the development of later build systems:

• Make is most often used by C and C++ developers, even though it does, in theory, work for other languages.

• Makefile instructions are written as shell commands, which are inherently platform-specific.

• Make does not deal well with files spread across a complex directory structure, such as often the case in Java and many other languages and platforms.

Java Build Tools

Ant

Apache Ant, begun in 2000, is a popular automated build tool that was designed explicitly to address some of the shortcomings of Make for Java builds.

Ant (continued)

Key features:

• open source

• build scripts written in XML in a file named build.xml

• supports same basic functionality as Make, but…

• is platform-agnostic, unlike Make

• contains a built-in recognition of common steps in building Java projects, unlike Make

• integrates with unit testing frameworks, such as JUnit

Ant (continued again)

See an example of an Ant build.xml file.

Ant (again once more)

Ant has suffered criticism on a few fronts:

• XML, today, is considered to be overly verbose and redundant as a structured data markup language. JSON and YAML are considered more concise and elegant.

• Each build script tends to be custom: there is little convention applied to scripts. This results in “reinventing the wheel” each time.

• Ant suffers from legacy Java compatibility issues: some tasks that Ant supports, such as javac and java, use default argument options that are no longer current with the latest Java. Changing the implementation of these would break older code, so the developers have not updated Ant to more modern usage conventions.

Maven

Apache Maven is a successor to Apache Ant, initially released in 2004, that aims to address some of Ant’s shortcomings, plus adding some additional features.

Maven (continued)

Key features:

• open source

• build scripts written in XML in a file named pom.xml

• supports same basic functionality as Ant, but…

• unlike Ant, Maven takes a “convention over configuration” approach

• supports multiple languages, not just Java (although in reality it is mostly used for Java)

• plug-in architecture allows 3rd-party tools to be integrated into Maven builds

• maintains a repository of plug-ins for easy discovery and a package manager to handle installing them

Maven (continued again)

See an example of a Maven pom.xml build file.

Maven (continued again once more)

Like build systems before it, Maven has its critics.

• As with Ant, Maven’s reliance on XML for configuration scripts seems outdated.

• Maven requires a rigid adherence to convention: developers requiring a custom configuration may find it difficult to work with.

• A project’s build steps are not well-documented when using Maven, since it relies so much on hidden defaults and doesn’t explicitly include each step in the build scripts..

Gradle

Gradle is [yet another] open source build automation tool primarily focused on Java projects.

Gradle

Key features:

• open source

• build scripts written in Groovy in a file named build.gradle

• supports same basic functionality as Maven, but…

• easier to customize a build than in Maven

Gradle (continued again)

See an example of a Gradle build.gradle file.

Intermission

Package managers

A “package” is software that has been designed to be easily distributed via a “package manager” (sometimes called a dependency manager) so that others can easily download, install, configure, and use the package in their own applications.

• Java has two popular package managers: Maven and Gradle.
• Javascript’s default package manager is npm.
• Python’s default package manager is pip
• Swift’s default package manager is the Swift Package Manager
• … etc.

Package managers thus maintain repositories of packages and provide the tools to install and configure those packages.

Now on with the show

As we’ll see, the Javascript and Python build tools ecosystems center heavily on package managers.

JavaScript Build Tools

Overview

JavaScript has one of the most sophisticated and rapidly evolving tooling environment of any language. Some popular build-related tools include:

• npm and yarn - package managers
• webpack and parcel - bundlers

Package managers

npm and yarn are package managers for Javascript that have build-related capabilities.

• npm is the default package manager for Node.js
• yarn is a newer package manager that aims to be faster and more secure than npm
• document, install, and manage dependencies and configuration settings in a package.json file
• installing a project’s dependencies is as simple as running npm install
• can also include arbitrary scripts that can be run with npm run <script-name> or yarn run <script-name>
• startup script is usually called with a simple npm start or yarn start
• test scripts are usually run with a simple npm test or yarn test
• see example of a package.json file

Webpack and parcel

webpack and parcel are bundlers for Javascript.

• traditionally take a bunch of separate Javascript files and bundle them into a single file.
• can also be used as a task runner to trigger other actions like minification, transpilation, etc.
• include “hot reloading” which means that whenever you save a file, a new build will be triggered automatically
• webpack is the more common bundler in use today, but configuring it is complex
• parcel markets itself as “zero-configuration” competitor

Python Build Tools

Overview

The Python build tool ecosystem has historically been less developed, but has recently taken inspiration from Java and JavaScript. Some popular tools:

• pip - package manager
• venv - virtual environments
• conda - package manager and virtual environments!
• pipenv - modular reproducible sharable virtual environments
• distutils and setuptools - build tools for creating packages

Package managers and virtual environments

pip is the default package managers for Python. pip is used to document, install, and manage dependencies in most Python projects

• packages in pip are stored in the Python Package Index (PyPI, a.k.a “Cheeseshop”) - a repository of Python packages
• venv - allows dependencies to be installed modularly in a project-specific “virtual environment”
• pipenv - makes it easier to manage virtual environments in a portable, easily-shareable way

conda is a competitor package manager like pip that also manages virtual environments like venv and pipenv. It is part of the popular Anaconda data science toolkit.

Distutils and setuptools

When it comes to creating packages so that software is easily distributed from the PyPI via pip or conda, Python has two main tools:

• distutils - the default module for creating packages for distribution via a package manager
• setuptools - a higher-level tool that builds on top of distutils and provides additional functionality
• build - an even higher-level build tool that works with projects designed for use with setuptools as well as other build systems.

In practice, one need only use build directly, as this provides a high-level interface to the functionality of setuptools and distools - see discussion on this point.

twine is a tool for publishing packages to the PyPI repository so others can access, install, and use them.

Python’s build ecosystem (continued)

Python builds result in packages consisting of two artifacts:

• sdist - a compressed archive of the source code that can be used to build the software on an end-users machine
• wheel - the pre-built distribution package (a.k.a. “eggs”) that can be installed directly without requiring the end-user to build the software

Conclusions

You have seen some of the most significant software build tools and how they have evolutionarily progressed throughout build system history.

• Thank you. Bye.