Know your shell - A deep dive into the whys and hows

Shells expose a way to communicate directly with the operating system. Often, shell programming languages have terse syntax because they're designed to be typed in frequently into your terminal. Which is probably why we have mv and not move as well as rm instead of a verbose remove. If you're coming from a Unix based OS, the terminal is usually the default way to do things, unlike Windows OS which prioritizes GUI experiences for most things.

In this post, we'll look at some of points to keep in mind to be able to write shell scripts like a true power user.

This post assumes you're in a Unix like environment, I have very little experience with how scripting behaves on Windows machines.

The language of the shell is interpreted not compiled

Shell scripting languages such as fish and bash exist and are the default syntax you would use for most tasks. However, that doesn't take full of advantage of what the shell can do. Any (if not, then most) interpreted language can be invoked directly from the shell as a script. This is made possible because of the 'shebang' directive which simply tells the operating system which program to invoke when parsing the contents of the script.

#! /usr/bin/env bash

This means that you can use JavaScript or Python, for example, as the language to write your scripts in which can be helpful for particularly complex routines that would simply be too ugly in something like bash.

Here's an example using JavaScript and the Bun runtime

File descriptors and stream redirection

Standard input,output and error can be manipulated through file descriptors and redirection symbols. This is helpful for accessing any of these streams from any point in your shell program. For example let's say that you want to run a development server and open Neovim in the same terminal tab. The natural thing would be to just add an ampersand at the end of the command to mark that job as background:

However, this doesn't cut it in this case because when the server reloads, it will dump its output into the current window which isn't what you want:

The proper way to do it would be to redirect stdout to /dev/null which is like a blackhole for the operating system because what goes there is never found again. The only downside may be that the server may need to write to disk and that output will instead be discarded. Use it carefully.

Stream redirection just allows us to specify where to read from (stdin) or where to write to (stderr/stdout). Nothing fancy 🙂

Environment variables define the state

Environment variables are used a lot in programming, they help us swap values between for example, development and production API keys. They also help set predefined/default values which can be used by other programs.

Example use cases for environment variables are:

• which editor to invoke e.g when running git commit with no -m flag
• what path to use as the base configuration path for your programs e.g XDG_GONFIG_HOME.

We can see the list of current environment variables by typing the following command:

Job automation for routine tasks

The beauty of automation lies in the fact that you can focus on other things and perhaps just keep track of how your script is exiting (either successfully or with errors). Most of the time crontab is the way to achieve this, which is a daemon (background process) that executes specified instructions at set intervals.

Any task that has an expected outcome and defined steps can be automated. The programmer's creativity is usually the limit to what can be automated.

Here's an example of using the crontab utility:

Accessing system information

When writing scripts certain system information is important to determine the next step to take, for example, architecture type in installation scripts so as to download the correct target binary.

Knowing this kind of information can increase the flexibility of your programs:

Function arguments (argv) & custom shell functions

argv is a list of arguments that is passed to a program when it is invoked. It is useful when writing shell functions or when we wish to access positional arguments. argv is a list, which means we can iterate (loop) over it as well.

Here's an example in fish:

Shell functions work just as you would expect from other programming languages albeit the syntactical differences. Instead of wrapping arguments in parentheses we use flags and positional arguments to specify how we pass input to the function. Flags can either be long (usually denoted with a double dash or --some-flag) or short (which is just a shortened flag of the variable which looks like -s). If the flag expects a value, then it has to be passed immediately after the flag.

Process management

Commands like ps, kill and disown help us manage current processes. You can use them for things like deleting rogue processes, seeing which processes are hogging the system resources just to mention a few. Let's say I created a background task and now wish to kill it. Here's one way to do it:

The top command is also available for real-time system monitoring but I like btop, which is a prettier alternative to the former utility.

Keybindings for commonly used programs and behaviors

Navigation should be simple. The moment navigation becomes a bottleneck you become tired from monotonous routines which is a productivity killer.

Different shells have different ways of handling key mapping. I find the fish syntax to be more concise and readable. The choice for which modifiers to use depends on personal preference, I like using Alt instead of Ctrl (for ergonomic reasons with my current keyboard layout).

Here's my setup using fish:

Learn a few coreutils

It's surprising how much you can achieve by just using the GNU coreutils. You can sort of build an entire app from those utilities, and the best part is that they are battle tested. With time, muscle memory builds and it feels impossible to live without them anymore. Of course, you don't have to cram all that syntax at once but you can use a few documentation utilities

Piping and short circuiting

We can pass the output of a function as the input of another function. Think of how compose() functions work in FP, with each function call returning its result as input for the function above it:

Daemons for self booting services

Let's say we have a task we wish to run everytime the system boots up. This is where daemons come in handy.

The term has nothing cult about it. It just means a background running process that is not designed to be directly interactive with the user. I tried making a daemon in this post

Escape sequences and control characters

Conclusion

This was only a fraction of some things to keep in mind when working with shells. Experiences vary and as far as I have learnt, these things are some of the concepts that helped me look at shell scripting differently. You can checkout this GitHub Gist for more resources on shell scripting.