8  Getting comfortable on the Command Line

A very common pattern in Linux administration is to read a log file to look for errors or clues to an error, adjust a configuration setting as a result, and then restart a process to pick up the changes.

Many of the files you’ll interact with as a Linux admin are just text files. In particular, application configuration files and logs are usually just text files.

You’ll find that your skills in understanding the Linux file tree, moving around, and seeing what’s in directories will be very helpful in getting to the files. Once you’re there, it’ll be useful to know how to actually read, write, copy, move, and delete files.

8.1 The structure of bash commands

bash and its derivatives provide small programs that each do one small thing well called a command.

A command is the program you want to run, usually an abbreviation of the word for what you want to do. For example, the command to list the contents of a directory is ls.

Arguments tell the command what to run on. They come after the command with a space in between. For example, if I want to run ls on the directory /home/alex, I can run ls /home/alex on the command line.

Many commands have default arguments. For example, ls runs by default on the directory I’m currently in. So if I’m in /home/alex, running ls and running ls /home/alex would return the same thing.

Options or flags modify how the command operates. Flags are denoted by having one or more dashes before them. For example, the ls command, which lists files, has the optional flag -l, which indicates that the files should be displayed as a list.

Flags always come in between the command and any arguments to the command. So, for example, if I want to get the files in /home/alex as a list, I can run ls -l /home/alex or navigate to /home/alex and run ls -l.

Some flags themselves have arguments. So, for example, if you’re using the -l flag on ls, you can also use the -D flag to format the datetime when the file was last updated.

So, for example, running ls -l -D %Y-%m-%dT%H:%M:%S /home/alex will list all the files in /home/alex with the date-time of the last update formatted in ISO 8601 format (which is always the correct format for dates).

It’s nice that this structure is standard. You always know that a bash command will be formatted as <command> <flags + flag args> <command args>. The downside is that having the main argument come all the way at the end, after all the flags, can make it really hard to mentally parse commands if you don’t know them super well.

Because there can be so many arguments, bash commands can get long. Sometimes you’ll see bash commands split them over multiple lines. You can tell bash you want it to keep reading after a line break by ending the line with a space and a \. It’s often nice to include one flag or argument per line.

For example, here’s that ls command more nicely formatted:

> ls -l \
  -D %Y-%m-%dT%H:%M:%S \
  /home/alex

This is at least a little easier to parse. There is also help available!

All of the flags and arguments for commands can be found in the program’s man page (short for manual). You can access the man page for any command with man <command>. You can scroll the man page with arrow keys and exit with q.

If you ever can’t figure out how to quit, ctrl + c will generally quit from any command line situation on Linux, Mac, and Windows.

Symbol	What it is
man	manual
q	Quit man pages (and many other situations)
\	Continue command on new line

8.2 Linux directories and files

In Linux, directories define where you can be. A directory is just a container for files and other directories.

In Linux, the entire file system is a tree (or perhaps an upside-down tree). The root directory, / is the base of the tree and a / in between two directories means that it’s a sub-directory. So the directory /home/alex is the alex directory, which is contained in home, which is in the root directory /.

In Linux, every directory is a sub-directory of / or a sub-directory of a sub-directory of / or…you get the picture. The sequence of directories that defines a location is called a file path.

Every Linux command happens at a particular file path – called the working directory. In some cases the commands you’re allowed to run or what they do will vary a lot based on where you are when they run.¹

File paths can be either absolute – specified relative to the root – or relative to the working directory. Absolute file paths always start with / so they’re easy to identify.

Depending on what you’re doing, either absolute or relative paths make more sense. In general, absolute file paths make more sense when you want to access the same resource regardless of where the command is run, and relative file paths make more sense when you want to access a resource specific to where you run it.

At any time, you can get the full path to your working directory with the pwd command, which is an abbreviation for print working directory. When you’re writing out a file path, the current working directory is at ..

Going back to the ls command, you can now see that the default argument to ls is .. You can test this for yourself by comparing the output of ls and ls .. They should be identical.

Aside from / and ., there are two other special directories.

.. is the parent of the directory you’re in, so you can move to the parent of your current directory using cd .. and to it’s parent with cd ../...

~ is the home directory of your user (assuming it has one). We’ll get more into what that means in a bit.

How does / compare to C:?

If you’re a Windows person, you might think this is analogous to C:. You’re not wrong, but the analogy is imprecise.

In Linux, everything is a sub-directory of /, irrespective of the configuration of physical or virtual drives that houses the storage. Frequently, people will put extra drives on their server – a process called mounting – and house them at /mnt (short for…you guessed it).

That’s different from Windows. In Windows you can have multiple roots, one for each physical or logical disk you’ve got. That’s why your machine may have a D: drive, or if you have network shares, those will often be on M: or N: or P:.

Along with being able to inspect directories, it’s useful to be able to change your working directory with the cd command, short for change directory.

Recap of commands in this section

Command	What it does/is	Helpful options	Example
/	system root
.	current working directory
ls	list objects in a directory	-l - format as list -a - all (include hidden files)	$ ls . $ ls -la
pwd	print working directory		$ pwd
cd	change directory		$ cd ~ / D ocuments
~	home directory of the current user		$ ls ~

8.3 Reading text files

Being comfortable opening and navigating around text files is an essential IT/Admin skill.

cat is the command to print a file, starting at the beginning.

Sometimes you’ve got a really big file and you want to see just part. less starts at the top with the ability to scroll down. head prints the first few lines and quits. It is especially useful to peer at the beginning of a plain text data file (like csv) as it prints the first few rows and exits – so you can preview the beginning of a very large data file very quickly.

tail skips right to the end of a file. Log files usually are written so the newest part is last – so much so that “tailing a log file” is a synonym for looking at it. In some cases, you’ll want to tail a file as the process is still running and writing information to the log. You can get a live view of the end of the file using the -f flag (for follow).

Sometimes you want to search around inside a text file. You’re probably familiar with the power of regular expressions (regex) to search for specific character sequences in text strings. The Linux command to do regex searches is grep, which returns results that match the regex pattern you specify.

The true power of grep is unlocked in combination with the pipe. The Linux pipe operator – | – takes the output of the previous command and sends it into the next one.

Haven’t I seen the pipe before?

The pipe should feel extremely familiar to R users.

The {magrittr} pipe, %>%, has become extremely popular as part of the tidyverse since its introduction in 2013. A base R pipe, |>, was released as a part of R 4.1 in 2021.

The original pipe in {magrittr} took inspiration from both the Unix pipe and the pipe operator in the F# programming langauge.

A combination I do all the time is to pipe the output of ls into grep when searching for a file inside a directory. So if I was searching for a file whose name contained the word data, that might look something like ls ~/projects/my-project | grep data.

Command	What it does	Notes + Helpful options
cat	Prints a file.
less	Prints a file, but just a little.	Can be very helpful to look at a few rows of csv. Lazily reads lines, so can be much faster than cat for big files.
tail	Look at the end of a file.	Useful for logs, where the newest part is last. The -f flag is useful for a live view.
head	Look at the beginning of a file.	Defaults to 10 lines, can specify a different number with -n <n>.
grep	Search a file using regex.	Writing regex can be a pain. I suggest testing expressions on regex101.com. Often useful in combination with the pipe.
|	the pipe

8.4 Copying, moving and removing files and directories

You can copy a file from one place to another using the cp command. cp leaves behind the old file and adds the new one at the specified location. You can move a file with the mv command, which does not leave the old file behind.

If you want to remove a file entirely, you can use the rm command. There is also a version to remove a directory, rmdir.

Warning

Be very careful with the rm command.

Unlike on your desktop there’s no recycle bin! Things that are deleted are instantly deleted forever.

If you want to make a directory, mkdir makes a directory at the specified filepath. mkdir will only work if it’s creating the entire file path specified, so the -p flag can be handy to create only the parts of the path that don’t exist.

Sometimes it’s useful to operate on every file inside a directory. You can get every file that matches a pattern with the wildcard, *. You can also do partial matching with the wildcard to get all the files that match part of a pattern.

For example, let’s say I have a /data directory and I want to put a copy of only the .csv files inside into a new sub-directory. I could do the following:

> mkdir -p /data/data-copy
> cp /data/*.csv /data/data-copy

Command	What it does/is	Notes + Helpful Options	Example
rm	remove – deletes i m mediately and p e rmantenly	-r - r e cursively remove e verything below a file path -f - force - dont ask for each file	$ r m - rf o ld_docs/ BE VERY CAREFUL WITH -rf
cp	copy
mv	move
*	wildcard
mkdir/ rmdir	make/ remove directory	-p - create any parts that don’t exist

8.5 Moving things to and from the server

It’s very common to have a file on your server you want to move to your desktop or vice versa.

It’s generally easier to move a single file rather than a whole bunch. The tar command turns a set of files or whole directory into a single archive file, usually with the file suffix .tar.gz. Creating an archive also does some compression when it creates the archive file. The amount depends on the content.

The tar command is used to both create and unpack (extract) archive files and telling it which one requires the use of several flags. I never remember them – this is a command I google every time I use it. The flags you’ll use most often are in the cheat sheet below.

Once you’ve created an archive file, you’ve got to move it. The scp command is the way to do this. scp – short for secure copy – is basically a combo of SSH and copy.² scp is particularly nice because it uses the syntax you’re used to from using cp.

Since scp establishes an SSH connection, you need to make the request to somewhere that is accepting SSH connections. Hopefully your server is accepting SSH connections and your laptop is not.

You’ll almost certainly have the experience at some point of being on your server and wanting to scp something to or from your laptop. You need to do the scp command from a regular terminal on your laptop, not one that’s already SSH-ed into your server.

Command	What it does	Notes + Helpful options
tar	c ompress/decompress file/directory	Almost always used with flags. Create is usually tar -czf <archiv e name> <file(s)> Extract is usually tar -x fv <archive name>
scp	Copy across ssh	Can use most ssh flags (like -i and -v)

8.6 Writing files on the command line

There will be many situations where writing into a text file will be handy while administering your server – for example, when changing config files. When you’re on the command line, you’ll use a command line tool for writing into those files – meaning you’ll only have your keyboard to navigate, save, and exit.

There are times when you want to make files or directories with nothing in them – the touch command makes a blank file at the specified file path.

You also may want to take some text and make it into a file. You can do this with the > command. >> does the same thing, but appends it to the end of the file. This works similarly to the pipe, |, where the output of the left-hand side is passed as the input to a file on the right-hand side.

A common reason you might want to do this is to add something to the end of your .gitignore. You can’t just type a word on the command line and have it treated like a string – so you may need the echo command to have something you type treated as a string.

For example, if you want to add your .Rprofile file to your .gitignore, you could do that with echo .Rprofile >> .gitignore.

Command	What it does	Notes
touch	Creates file if doesn’t already exist.	Updates timestamp to current time if it does exist
>	Overwrite file contents	Creates a new file if it doesn’t exist
>>	Concatenate to end of file	Creates a new file if it doesn’t exist

8.6.1 Command line text editors

There are two command line text editors you’ll probably encounter – both extremely powerful text editing tools: nano and vi/vim.³

You can open a file in either by typing nano <filename> or vi <filename>. Unfortunately for many newbie Linux Admins it’s extremely easy to get stuck inside a file with no hint of how to get out!

In nano there will be helpful prompts along the bottom to tell you how to interact with the file, so you’ll see once you’re ready to go, you can exit with ^x. But what is ^? On most keyboards, you can insert the caret character, ^, by pressing Shift + 6. But that’s not what this is.

In this case, the ^ caret is short for Ctrl on Windows and for Command (⌘) on Mac. Phew!

Where nano gives you helpful – if obscure – hints, vim leaves you all on your own. It doesn’t even tell you you’re inside vim!

This is where many people get stuck and end up having to just exit and start a new terminal session. It’s not the end of the world if you do, but knowing a few vim commands can help you avoid that fate.

One of the most confusing things about vim is that you can’t edit the file when you first enter. That’s because vim keybindings were (1) developed before keyboards uniformly had arrow keys and (2) are designed to minimize how much your hands need to move.

If you feel like taking the time, learning vim keybindings can make navigating and editing text (code) files easier. Plus it just feels really cool. I recommend spending some time trying. In this section, I’m just going to help you get the minimum amount of vim you need to be safe.

When you enter, you’re in normal mode, which is for navigating through the file. Typing things on your keyboard won’t type into the document, but will do other things.

Pressing the i key activates insert mode. For those of us who are comfortable in a word processor like Word or Google Docs, insert mode will feel very natural. You can type and words will appear and you can navigate with the arrow keys.

Once you’re done writing, you can go back to normal mode by pressing the escape key. In addition to navigating the file, normal mode allows you to do file operations like saving and quitting.

File operations are prefixed with a colon :. The two most common commands you’ll use are save (write) and quit. You can combine these together, so you can save and quit in one command using :wq.

Sometimes you may want to exit without saving. If you’ve made changes and try to exit with :q, you’ll find yourself in an endless loop of warnings that your changes won’t be saved. You can tell vim you mean it with the exclamation mark and exit using :q!.

Vim Command	What it does	Notes + Helpful options
^	Prefix for file command in nano editor.	Its the ⌘ or Ctrl key, not the caret symbol.
i	Enter insert mode in vim
escape	Enter normal mode in vim.
:w	Write the current file in vim (from normal mode)	Can be combined to save and quit in one, :wq
:q	Quit vim (from normal mode)	:q! quit without saving

8.7 Comprehension Questions

1. If you don’t know the real commands for them, make up what you think the bash commands might be to do the following. So if you think you’d create a command called cmd with a flag -p and an argument arg, you’d write cmd -p <what p does> <arg>. In the next chapter you’ll get to see how close you got to the real thing:

   1. Change Directories, the only argument is where to go

   2. Making a Directory, with an optional flag to make parents as you go. The only argument is the directory to make.

   3. Remove files, with flags to do so recursively and to force it without checking in first. The only argument is the file or directory to remove.

---

1. It’s interesting to note that this is also true on your computer - when you open a program, a particular user is running a program on your computer and opening a GUI window for you to interact with. The point-and-clicking obfuscates this, but it’s still true.

2. It’s worth noting that scp is now considered “insecure and outdated”. The ways it is insecure are rather obscure and not terribly relevant for most people. But if you’re moving a lot of data, you may want something faster. If so, I’d recommend more modern options like sftp and rsync. I probably wouldn’t bother if you’re only occasionally scp-ing small files to or from your server.

3. vi is the original fullscreen text editor for Linux. vim is its successor (vim stands for vi improved). For our purposes, they’re completely interchangeable.