knowledge-kitchen

Bash Scripting - The Bones of Automation

Automate your life.

  1. Overview
  2. Script File Setup
  3. Built-In Commands
  4. Programming
  5. Grouping
  6. Exit codes
  7. Automation
  8. Conclusions

Overview

Concept

bash is the most commonly used shell for Unix/Linux

Value

Of what use to us is a 45-year-old set of rudimentary operating system commands?

Interfaces

bash commands can be run in one of two ways:

Script File Setup

Shebang

The very first two characters in a bash script file must be shebang #!

Follow this with the path name to bash, e.g. (your path may differ)

#!/bin/bash

You can also usually find the location of bash by referring to your machine’s environment settings

#!/usr/bin/env bash

Anytime a shebang is used, the kernel will pass the path to the script as the first argument to the script. Yes, scripts can take arguments.

Try it!

File Permissions

Give yourself execute permissions to run a script file directly:

chmod u+x file.sh

Now execute the script:

./file.sh

If you don’t have execute permission, but do have read permission, you can still run the file:

bash file.sh

Built-In Commands

All commands

Bash comes with a bunch of built-in commands. To show a list of them all:

enable | cut -d' ' -f 2

All keywords

There are also a few reserved keywords you can review:

compgen -k

PATH Variable

bash’s PATH variable contains a list of directories where bash will, by default, look for executable programs and shell commands.

Display the contents of PATH:

echo $PATH

Add a directory to the PATH… directories must be separated by colons.

PATH=$PATH:/Users/foo/Downloads/spyware

Time-Keeping

The time command can be useful to determine how long a process take to complete. It reports:

Measure the time taken by the ls -la command:

time ls -la

Try it!

Pipes

Pipes allow the output of one program to serve as input for another.

List only .txt files in the current working directory:

ls -l | grep "\.txt$"

Swap all vowels in a file listing with ‘u’

ls -l | sed -e "s/[aeio]/u/g"

If you really want to be a devops guru, learn the grep, sed, and awk commands in detail - here’s a decent intro.

Output redirection

While the system default output is to print to the command line, it is possible to redirect any command’s output to a file or other resource.

Output the list of files from the previous example to a file named long_listing.txt:

ls -l | grep "\.txt$" > long_listing.txt

Append the word, flibbertigibbet to the file named will-o-the-whisp.txt

echo "flibbertigibbet" >> will-o-the-whisp.txt

Ignore a cry for help by redirecting it to the null device: /dev/null:

echo "Help\!\!\!" > /dev/null

Note we need to esape the exclamation points, since ! has special meaning.

Command substitution

It is possible to use the output of any bash command as data within a bash script.

Place $() around the command whose output you wish to capture

# output the text, "Hello world" the hard way
RECIPIENT=$(echo world)
echo Hello $RECIPIENT!

Try it!

Programming controls

Variables

Variable assignments must not have spaces around the = sign:

some_variable_name="this is a string"

Read value of a variable with a $ in front of variable name

echo the value of some_variable_name is $some_variable_name

Aliases

The alias command allow you to set other names for common commands.

alias ll="ls -l"

To execute a command alias, in this case to perform ls -la:

ll

See all aliases currently set in the shell session:

alias

Conditionals

bash supports the usual if/else if/else controls, albeit with archaic syntax:

#!/usr/bin/env bash

T1="foo"
T2="bar"

if [ "$T1" == "$T2" ]; then
    echo expression evaluated as true
else
    echo expression evaluated as false, obviously
fi

Try it!

Loops

bash supports both for and while loops:

For:

#!/usr/bin/env bash

for i in {1..5}; do
  echo $i
done

While:

#!/usr/bin/env bash

COUNTER=0

# note the archaic syntax for the less than operator
while [  $COUNTER -lt 10 ]; do
	echo The counter is $COUNTER
	let COUNTER=$COUNTER+1
done

Loops (continued)

It is also possible to loop through lines of output from another command using command substitution:

for i in $( ls ); do
    echo ... $i ...
done

while IFS= read -r line; do
    echo "... $line ..."
done <<< $(ls)

Try it!

Functions

A function with a local variable:

#!/usr/bin/env bash
HELLO=Hello
function hello {
        local HELLO=World
        echo $HELLO
}
echo $HELLO
hello
echo $HELLO

A function with an argument:

#!/usr/bin/env bash
function e {
    echo $1
}
e Hello
e World

Exports

A copy of a variable defined within a script can be copied into your command-line environment, which makes it available to any other scripts you run

export some_variable_name

Functions can also be exported

export -f some_function_name

Print what has been exported to the command-line:

export

Exit codes

Concept

Bash scripts can and should pass exist codes to their parent processes when they complete.

The $? variable

Here we, for example, attempt to move a file and indicate whether it was a success or not, by checking the value in the $? variable.

#!/usr/bin/env bash

# attempt to move a file, for example...
mv foo.txt /foo/bar/baz.txt

# check whether it worked.
if [ $? -eq 0 ]
then
  echo "Successfully moved file"
  # we could do any relevant follow-up actions here
else
  echo "Could not move file"
  # we could do any relevant follow-up actions here
fi

Command chaining

Commands can be chained together using boolean logic operators.

npm run lint && npm run test-unit && npm run test-casper-runner

Command chaining (continued)

Or (||) logic can also be useful in command chaining.

./tmp.sh && echo "bam" || (sudo ./tmp.sh && echo "bam" || echo "fail")

Custom exit codes

You can, of course, specify exit codes at relevant places in your own bash scripts using the exit keyword.

#!/usr/bin/env bash

# attempt to move a file, for example...
mv foo.txt /foo/bar/baz.txt

# check whether it worked.
if [ $? -eq 0 ]
then
  echo "Successfully moved file"
  # we could do any relevant follow-up actions here
  exit 0
else
  echo "Could not move file"
  # we could do any relevant follow-up actions here
  exit 1
fi

Grouping

Parentheses

Parentheses

Commands grouped within parentheses are separate processes. Variables defined within the sub-process are not shared with the parent process.

a=1
(
a=2
)
echo $a
# prints 1

Braces

Commands grouped with braces do not spawn a sub-process, so variables therein are shared:

a=1
{
a=2
}
echo $a
# prints 2

Automation

Startup Scripts

Bash is used for everyday automation by developers:

Place whatever bash commands you want in files by either of these names in your user account home directory in order to take advantage of this feature.

Aliases File

Many developers use bash aliases for common commands.

For good housekeeping, store many aliases, if you have them, in a ~/.bash_aliases file and load them automatically in your .bashrc script every time a shell session starts.

Example .bashrc script to check whether such an aliases file exists and execute it, if so:

if [ -f ~/.bash_aliases ]; then
. ~/.bash_aliases
fi

Software Development Automation

Contemporary software developers depend upon automation to simplify and speed up many pedestrian tasks.

Continuous integration example

A fragment a script that could be used in Jenkins, a popular automation tool, to execute some follow-up actions anytime a change is committed to particular branches of a version control repository:

#!/usr/bin/env bash

if [[ $BRANCH_NAME == "main" ]] || [[ $BRANCH_NAME == "main_dev" ]]
then
    ./runUnitTests.sh $REPOSITORY_NAME $BASE_BUILD_CORE $BRANCH_NAME $BUILD_NUMBER || echo "The npm may fail but the report exists"
fi

Jenkin automatically sets up variables named BRANCH_NAME, REPOSITORY_NAME, etc and this custom code uses those to decide whether to run an automated testing script.

Continuous deployment example

Travis CI, another popular automation tool, can run any arbitrary bash script when the changes are committed to the codebase.

deploy:
  provider: script
  script: bash scripts/deploy.sh
  on:
    branch: main

In this case, we see the contents of a YAML settings file named .travis.yml that would be placed in the version control repository. This file informs Travis CI to execute a script named deploy.sh anytime new code is pushed to the main branch. Typically, such a file would log into a remote server via ssh, copy the contents of the repository to the server, build the code, and execute it.

NPM script example

All Node.js projects using the Node Package Manager (NPM) contain a package.json file that can define a set of labeled bash scripts with names.

"scripts": {
    "start": "node server.js",
    "start-dev": "nodemon ./server.js",
    "test-unit": "mocha ./test/unit --exit",
    "test": "npm run lint && npm run test-unit && npm run test-casper-runner",
    "lint": "eslint .",
    "codecov": "npm run test && (codecov || true)",
    "autofix": "eslint --fix .",
    "validate": "npm ls"
  }

Such scripts can then be easily executed from the command line by referring to their labels.

npm run test

Education automation

Your professor may even use bash scripts to automatically parse git logs to gauge the level of code activity of each student.

# loop through each contributor
git log --format='%aN' | sort -u | while read user
do
	# show each contributor's stats
    printf '%-25s' "- $user -"
    echo -n $(git log --shortstat --author="$user" --after="$AFTER_DATE" --before="$BEFORE_DATE" | grep -E "Merge" | awk '{merges+=1} END {printf "merges: %03d | ", merges}')
    echo -n $(git log --shortstat --author="$user" --after="$AFTER_DATE" --before="$BEFORE_DATE" | grep -E "fil(e|es) changed" | awk '{commits+=1; files+=$1; inserted+=$4; deleted+=$6} END {printf " commits: %05d | additions: %05d | deletions: %05d | files: %05d", commits, inserted, deleted, files }')
    echo ""
done

Fork it and try it!

Conclusions

You are now a beginner bash programmer.