Statements: For

Writing for statements with no unintended side effects is probably the hardest part of writing shell scripts. That was my opinion, until now…

Using nullglob to loop over files

Let’s say you want to print files in a directory:

for file in directory/*
do
   printf "%s\n" "${file}"
done

This works, if you have some files in the directory. If not, you get literally the output “directory/*. Getting this unexpanded string back is extremely useless. Consulting Stack Overflow or - if all else fails - the documentation, we fix this to read:

shopt -s nullglob
for file in directory/*
do
   printf "%s\n" "${file}"
done
shopt -u nullglob

That code is fine… unless you later change the printf line to call another function, that did not expect nullglob to be set. This is an easy mistake to make, so I’d expect, that I will make it. The actual proper solution is:

shopt -s nullglob
for file in directory/*
do
   shopt -u nullglob
   printf "%s\n" "${file}"
done
shopt -u nullglob

This is correct, because the bash expands all items of the for loop during the interpretation of the for line and not again afterwards. It is necessary to keep the final shopt -u nullglob, in case the directory is empty and no looping is done.

I find this quite ugly and it is a lot of type-work. A source of bugs, if you ask me.

Beauty in macros

Macros in bash ? Yes there is a macro facility in bash. It’s called alias and it’s available for scripting with shopt -s expand_aliases (bash) or setopt aliases (zsh). alias as a macro facility is very limited. It can not be used in the middle of a statement and it knows no arguments.

Note

That you can use aliases in shell scripts is something I just recently learned through the yoctu project. To me learning this, felt like a game changer.

Lets define these aliases (bash shown, for zsh see mulle-compatibility.sh):

alias .foreachfile="set +f; shopt -s nullglob; IFS=' '$'\t'$'\n'; for"
alias .do="do
set +f; shopt -u nullglob; IFS=' '$'\t'$'\n'"
alias .done="done;set +f; shopt -u nullglob; IFS=' '$'\t'$'\n'"

Now the loop can be rewritten to:

.foreachfile file in directory/*
.do
   printf "%s\n" "${file}"
.done

which is very nice looking and foolproof.

Looping over words

You expect a list of names and you want to run a loop over them. But some joker is actually passing you wildcards:

foo()
{
   local names="$1"

   for name in ${names}
   do
      printf "%s\n" "${name}"
   done
}
foo 'a * b'

The output includes the unintended list of files of the current directory

a
404.html
atom.xml
_config.yml
css
fonts
Gemfile
Gemfile.lock
images
_includes
index.markdown
_layouts
modern-bash-scripting.sublime-project
modern-bash-scripting.sublime-workspace
_posts
_site
b

This is the so called “globbing” in effect, which is on by default. It can be turned on with set +f and off with set -f. Above code can be fixed with another macro alias .for="set -f; for:

foo()
{
   local names="$1"

   .for name in ${names}
   .do
      printf "%s\n" "${name}"
   .done
}
foo 'a * b'

Output:

a
*
b

Note

I find turning off globbing globally for the whole script, as a way to circumvent the problem, to be inconvenient. Any interactive shell has globbing enabled, and this would require me to remember to also selectively turn globbing on and off, to test my code in a copy/paste fashion.

Looping over a PATH

By setting the global variable IFS you can control the way the shell splits a string up as items for the for loop. Forgetting to reset IFS properly can be even more devastating then a missing nullglob.

IFS=':'
for x in a:b:c:d
do
   IFS=' '$'\t'$'\n'
   printf "%s\n" "$x"
done
IFS=' '$'\t'$'\n'

With the macro alias .foreachpath="set -f; IFS=':'; for" this can be beautified to:

.foreachpath x in a:b:c:d
.do
   printf "%s\n" "$x"
.done

Some more macros

Here are some more useful macros. Each specify a different IFS.

alias .foreachline="set -f; IFS=$'\n'; for"
alias .foreachitem="set -f; IFS=','; for"
alias .foreachcolumn="set -f; IFS=';'; for"