Once someone starts to become comfortable with using the command line in interactive mode, writing shell scripts to automate common tasks is a logical next step. The problem is that shells, as languages, are absolutely full of shortcomings and “gotchas.”

Here’s the Wikipedia definition of “gotcha”:

[A] gotcha is a valid construct in a system, program or programming language that works as documented but is counter-intuitive and almost invites mistakes because it is both easy to invoke and unexpected or unreasonable in its outcome

Writing scripts in Bash may often lead to scenarios where everything appears to work fine, until some edge case is reached that leads to unexpected behavior (I’m looking at you, filenames with spaces in them).

This was written half as a rant and half as a source of information. Hopefully, I’ll be able to link people to it in the future to explain why I dislike Bash. Even if that never happens, at least I got to practice converting frustration into text.

Expansion

I’ll start by talking about variable expansion, because expanding things wrong is probably the most-common beginner mistake in Bash that I’ve seen.

Let’s start by creating and assigning a variable (with whitespace in it, because I want to break things):

foo='bar baz'

Now, how do we actually use that variable? Do we need quotation marks, or not? Let’s look at our options.

Types of Quotation Marks

When it comes to quoting variables, you have three choices:

  • Double quotes
  • No quotes
  • Single quotes

(There’s also the grave accent, `, but don’t let it trick you: despite the visual similarity to a quote mark, it has nothing to do with quoting things in Bash)

You can also choose to use curly brackets: $foo vs ${foo}. The brackets are necessary for some things (e.g. parameter expansion), but in this case, they aren’t needed. I’m going to omit brackets when they aren’t necessary.

Double Quotes

"$foo" will expand the variable. When used as an argument to a command or function, it will pass the contents of the variable as a single argument. This is the “right” way of expanding a variable. Rule #1 of Bash: If you’re expanding something, put it in double-quotes.

Like every rule, this does have exceptions, but if you know them by heart already then you probably aren’t in this post’s target audience.

No Quotes

$foo acts similarly; it will also expand the variable…but it then splits it into multiple arguments at every whitespace* character if you’re using it as an argument to a command or function.

*Semi-complicated-technical-note: it doesn’t have to be whitespace; it just is by default. Bash actually uses the special IFS variable to determine which characters will split variables. The default is spaces, tabs, and newline characters.

Single Quotes

'$foo' is a literal string: This produces the text “$foo”, not the contents of the foo variable. No expansion will occur inside single quotes.

How Expansion Can Cause Problems

So, different quotation marks do different things. Now I’ll show you how things can break if you use them wrong.

Omitting Double Quotes

Okay, so we can put variables in double quotes, or not. That’s confusing. Two similar things that may seem to be the same…until something breaks.

#!/usr/bin/env bash

first='testfile'
rm $first # This does what I want

second='filename with spaces'
rm $second # This doesn't

The first example works fine: there isn’t any whitespace in the filename. The second example is a different story. Obviously, it’s supposed to delete the file filename with spaces, but it doesn’t. Since I neglected to put $second in quotes, it gets split on whitespace and expanded to three arguments, not one. So rm will try to remove the files filename, with, and spaces. Instead of deleting the file I wanted to delete, I just deleted three files that I didn’t want to delete. Oops. Put your variables inside double quotes.

Using Unassigned (or Empty) Variables

Alright, now what if you try to use a variable that hasn’t had anything assigned to it? Personally, I think it’d make sense for that to be an error that stops the program. I guess the Bash developers thought otherwise.

In C, or probably any compiled sane language, you have to declare your variables in addition to assigning a value to them.

int main(void) {
	int foo; // Declare in one statement
	foo = 1; // Assign in another

	int bar = 2; // Or just do both simultaneously
}

If you try to use a variable that hasn’t been declared, the compiler will give you an error and tell you to fix your horrendous mistake. An error may also occur (in Rust, for instance) if you try to use a variable that hasn’t had a value assigned to it.

In Bash? Unless you’re using associative arrays (which are arrays with text as indices instead of integers), you don’t need to actually declare anything. Misspell a variable name? You’ll just end up with an empty string.

#!/usr/bin/env bash

food=pizza
echo "$fod" # Oops. It'll run, though

Now let’s look at a real-world example from the Linux version of Steam to show how bad expanding to an empty string can get:

Moved ~/.local/share/steam. Ran steam. It deleted everything on system owned by user.

That is a terrifying title for a GitHub issue. Here’s what went wrong:

rm -rf "$STEAMROOT/"*

That was* in the script that Steam runs. It has a directory in a variable and tries to delete everything in that directory. But what happened when the STEAMROOT variable was empty? First, Bash expanded "$STEAMROOT/"* to /*. Then Bash expanded /* to every directory in /. Then rm deleted every file that the user had the ability to delete.

This went from “Delete Steam’s files” to “Delete every file I have.” Here’s how this could’ve been prevented:

if [[ -n "$STEAMROOT" ]]; then # Verify "$STEAMROOT" isn't null or empty
	rm -rf "$STEAMROOT/"*
fi

It’s that simple. One conditional check would’ve stopped Valve’s end-users from having their files deleted. If this doesn’t drive home the importance of making sure your scripts truly work, regardless of any edge cases (“Eh, why would anyone move their Steam directory?”), then I honestly don’t think anything will.

* Past-tense because the issue has (thankfully) been resolved

Using Single Quotes When You Shouldn’t Be

Here’s a fictitious quote from a hypothetical Bash novice:

please help my bash program does not work when i use echo '$variable'

―Someone in #bash on freenode, probably

By now, you should realize that single quotes probably aren’t what that hypothetical user needs to be using. Did I say put your variables inside double quotes yet?

Expanding Arrays

In case the rules for expanding normal variables weren’t confusing enough, there’s also the rules for expanding arrays!

#!/usr/bin/env bash

# Declaring an array containing three integers
myarray=(1 2 3)

How do we use that? Well, we can use the same syntax as when we expand a normal variable, but this will expand to the value of index 0 rather than the whole array:

#!/usr/bin/env bash

myarray=(1 2 3)

# Bad: don't do this with arrays
echo "$myarray" # Expands to 1
echo "${myarray}" # Also expands to 1

And even if you do want index 0, that’s still not the “right” way to do it. Here’s how you should access individual array values:

#!/usr/bin/env bash

myarray=(1 2 3)

# Good: the correct way to get the first item in an array
echo "${myarray[0]}" # Expands to 1

# And, of course, for the other items:
echo "${myarray[1]}" # Expands to 2
echo "${myarray[2]}" # Expands to 3
echo "${myarray[3]}" # Expands to an empty string

And remember how I mentioned curly brackets aren’t necessary with normal variable expansion? Well, they are necessary with arrays.

#!/usr/bin/env bash
myarray=(1 2 3)
echo "$myarray[1]" # This expands to '1[1]', not '2'

At least the quotation rules from before didn’t change:

#!/usr/bin/env bash

myarray=('foo bar' 'baz')

rm "${foo[0]}" # Tries to remove the file 'foo bar'
rm ${foo[0]} # Tries to remove the files 'foo' and 'bar'

So how do we use the whole array? That’s where things get fun. Instead of an index number, you use either @ or *. I don’t know why they picked those characters. * kinda makes sense, but @? Really?

Like with normal variables, you can either use double quotes or no quotes. This gives us four options:

  • @ and double quotes (spoiler alert: you probably want this one)
  • @ and no quotes
  • * and double quotes
  • * and no quotes

${myarray[*]} and ${myarray[@]} do the same things. A little robot birdie told me you shouldn’t use them. I guess the “use double quotes” rule still applies to arrays.

"${myarray[@]}" expands to each item in the array, with each item being its own argument. This is probably the one you want.

"${myarray[*]}" expands to each item in the array as a single string, with each item separated by the first character of the IFS variable.

#!/usr/bin/env bash

myarray=('foo bar' baz)

# This prints 'foo bar' on one line and 'baz' on the next
# This is correct
for word in "${myarray[@]}"; do
	echo "$word"
done

# This prints 'foo', 'bar', and 'baz' on separate lines
# This defeats the purpose of using an array
for word in ${myarray[@]}; do
	echo "$word"
done

Globs

Globs are used to select all files in a directory that match some pattern. For example, *.jpg will expand to all the filenames in the current directory that end in .jpg, if there are any. But if there aren’t any matches? In that case, the glob will not expand to anything; it will just stay as the literal string *.jpg.

ZSH will just do this:

$ ls *.jpg
zsh: no matches found: *.jpgs

Rather than run the command in an unintended way, it gives you an error and runs no commands.

This behavior can also be enabled in Bash with shopt -s failglob, but I think having it on by default like ZSH would be an improvement.

Types (or lack thereof)

Generally, programming languages have a number of types: strings, signed integers, unsigned integers, floats, arrays, booleans, dictionaries, etc. You’re probably able to create your own types as well, for example using structs and enums.

Bash only has three types: strings, ints, and single-dimensional arrays. Want a number with a decimal point? Too bad; you’ll probably have to store it as a string and pipe it to bc. Want an array of arrays? Something like myarray=( (1 2 3) (4 5 6) )? Nope, can’t do that either.

Return Values

Each functions in a normal programming language will probably return one of those types that I mentioned above, but—as I mentioned above—Bash is rather lacking when it comes to types. Bash is a shell, made to run commands, and its concept of “return values” makes this (painfully) obvious.

Exit Codes

Every command and function has an exit code. This is an integer ranging from 0-255 (inclusive). An exit code of 0 means the command completed successfully; any other exit code indicates that something went wrong somewhere. Exit codes are how Bash’s if statements work:

#!/usr/bin/env bash

if some_command; then # run some_command, and if it exits with 0...
	other_command	  # then run other_command
fi

some_command  # Or run some_command...
echo "$?"	  # and check its exit value with the "$?" variable

You may notice that 0 == true and everything else == false is the opposite of how C handles integers being used as booleans. I think it does make sense when you stop to think about it, but the problem is that you might have to stop and think about it: that’s just an additional cognitive load.

Text output

You’ll sometimes want to run commands and capture their output in variables. Bash has a thing called “Command substitution” for that:

#!/usr/bin/env bash

# Don't actually use ls like this
files=$(ls *.jpg) # Store the output of the ls command in the files variable

Oh, and I used ls in this example for a reason: it’s wrong. This may seem like an intuitive way to get a variable containing the names of the files in a directory, but it doesn’t actually work in practice. First, you’d want an array of filenames, not a normal string. Second, if you try to iterate over that variable with a for-loop, you’ll run into problems if there’s any whitespace (spaces, newlines, etc.) in any of the filenames.

Here’s how you’d do that correctly:

#!/usr/bin/env bash

# This could be any glob, really
pictures=(*.jpg)

# Now we can iterate over that
for file in "${pictures[@]}"; do # Quote the array...
	grep 'sometext' "$file"	  # and the variable
done

Scoping

Environment Variables

Let’s look at a technically valid (albeit slightly contrived) Bash script to print the first column of a filename that’s specified as an argument

#!/usr/bin/env bash
# print_first_column.sh

# Set PATH variable to first argument
PATH="$1"

# Run awk using that variable
awk '{print $1}' "$PATH"

And what happens when I run that?

$ ./print_first_column.sh hello_world.c
./print_first_column.sh: line 8: awk: command not found

That’s strange; I know I have awk installed. So, what happened? Well, PATH is the variable that lists the directories that Bash tries to run commands from. So when I re-assigned it with PATH="$1", I lost the ability to run commands without specifying their absolute paths.

Yep, you can just overwrite environment variables inside your script. The suggested way to to prevent this is to use all uppercase names for environment variables and all lowercase names for script variables (e.g. path="$1"), but it’d be nice if the problem didn’t exist in the first place.

Global Scope

I think that this somewhat ties my previous few complaints together.

In most languages, variables are only available inside the function they are created in. To use them in another function, they need to be passed to that function. Global variables are generally considered poor form, because it’s difficult to keep track of when and how they can be modified.

In Bash, variables are global by default: once a variable is created, it is accessible from anywhere, regardless of where it was created.

#!/usr/bin/env bash

makevar() {
	foo=1
	return 0
}

echo "$foo" # foo is undeclared and unassigned
makevar		# foo is declared and assigned in the makevar function
echo "$foo" # prints '1'

Bash does have a local keyword that can be used to declare variables that are only usable within the function they were created in (and its children), but this is opt-in rather than opt-out. Most scripts you find probably aren’t using it.

Variable assignment

This is a relatively small complaint, but I’ll mention it anyways. Here’s some C:

int main(void) {
	int foo;

	foo = 1;
	foo= 2;
	foo =3;
	foo=4;
}

As you can see, the only difference between those four assigments is the spaces around the = sign. In C, those are all valid.

They aren’t in Bash.

#!/usr/bin/env bash

foo = 1 # Wrong. Doesn't work.
foo= 2  # Also wrong. Also doesn't work.
foo =3  # Still wrong. Still doesn't work.
foo=4;  # Right

My best guess for why spaces aren’t allowed is that it makes it easier to tell “variable equals value” apart from “command with two arguments” (e.g. echo=foo vs. echo = foo) when Bash parses the script.

Since Bash is a shell and therefore (unlike most languages) has a heavy focus on running external commands, this may actually be a valid tradeoff. Regardless, I’m sure this has caused a few problems to beginners learning Bash— especially those beginners who are already familiar with other languages that do allow spaces there.

Conclusion

Alright, so I’ve pointed out plenty of areas where Bash could be more user-friendly. I’ve pointed out real examples of how poorly-written Bash scripts can cause problems.

Despite its problems, Bash isn’t going away any time soon (thanks, POSIX). The best thing you can do is get good at it so you don’t make those costly mistakes. For learning Bash, I strongly recommend GreyCat’s Wiki, especially the FAQ and Pitfall Guide.

Or you could just go learn Python, I guess.

comments powered by Disqus