7  The Command Line + SSH

The biggest difference between working on your desktop and working on a server is that servers generally do not have graphical user interfaces (GUIs).

If you want to adjust the system settings on your laptop or navigate from directory to directory, you can click through a file tree or open up your preferences pane. On a server, interaction is via the command line – an all-text interface where you type commands.

Once nice thing is that once you feel comfortable using the command line on a server, you’ll probably find that there are many things that will get easier for you locally as well! Plus you get to feel like a real hacker.

In this chapter we’ll walk through getting access to the command line on your computer, using it to connect to your remote server, and how to do basic interactions with a Linux system from the command line.

I’d recommend reading this chapter even if you’ve used SSH or the command line before. It’s probable you didn’t really understand why you took the steps you did. Hopefully, by the time you’ve finished this chapter you might have to google the exact commands, but you’ll never again confuse a public and a private key and what to do with each (we’ve all been there).

7.1 Setting up the command line

Before you start interacting with the command line on your machine, it’s helpful to have a mental model of what you’re interacting with.

There are three layers to interacting with the command line – the terminal, the shell, and the operating system commands.

The terminal is the visual program where you’ll type in commands. Depending on which terminal you’re using, the colors and themes available for the window, as will the options for having multiple tabs and panes, and the keyboard shortcuts you’ll use to manage them.

It is possible to spend A LOT of time customizing your terminal to be exactly what you like. Some might argue it is not be the best use of your time. Such people are no fun, and and having a terminal that’s super customized to what you like is great.

The shell is the program you’re interacting with as you’re typing in commands. It’s what matches the words you type to actual commands or programs on your system. Depending on which shell you choose, you’ll get different options for autocompletion, handy plugins, and coloring and theming of the actual text in your terminal.

There is some overlap of things you can customize via the terminal vs the shell, so mix and match to your heart’s content.

Lastly, the operating system is what actually runs the commands you’re typing in. So the set of commands available to you will differ by whether you’re using Windows or Mac or Linux.

graph LR
    A[A Human] --> |Types| B[Commands]
    A --> |Opens| E
    E[Terminal] --> |Opens| C
    C[Shell] --> |Dispatches| B
    D[Operating System] --> |Defines the set of| B
    D[Operating System] --> |Runs| B

In the next few sections of this chapter, we’ll get into how to set up your terminal and shell so that it looks and behaves exactly the way you want.

Note

I haven’t used a Windows machine in many years. I’ve collected some recommendations here, but I can’t personally vouch for them the way I can my Mac recommendations.

7.1.1 Choosing your terminal

The terminal is the graphical program you’ll interact with to open terminal windows, have tabs, and do some theming of the window itself.

If you’re using a Mac, you’ve got a built-in terminal app, conveniently called Terminal. It’s fine.

If you’re going to be using your terminal more than occasionally, I’d recommend downloading and switching to the the free [iTerm2](https://iterm2.com/), which adds a bunch of niceties like better theming and multiple tabs.

If you’re using Windows, there are a variety of alternative terminals you can try, but the built-in terminal is the favorite of many users. Experiment if you like, but feel free to stick with the default.

7.1.2 Choosing and configuring your shell

The default shell for MacOS (and Linux) is called bash. It’s a great shell, and it hasn’t been replaced in a very long time. Many people – including me – prefer to use other shells that build on top of bash, adding nicer theming and convenience features.

If you don’t already have a favorite bash alternative, I recommend zsh. It has a few advantages over bash out of the box, like better autocompletion. It also has a huge ecosystem of themes and plugins that can make your shell way prettier and more functional. There are plugins that do everything from displaying your git status on the command line to controlling your Spotify playlist.

Other popular alternative shells include Ksh and Fish.

The main reason to use zsh is the plugin ecosystem – so you’re going to want a plugin manager. The two most popular options are OhMyZsh and Prezto. I prefer and recommend prezto, but the choice is really up to you.

I’m not going to go through the steps of installing these tools – there are numerous online walkthroughs and guides that you can google.

But it is a little confusing to know what to customize where, so here’s the high level overview if you’ve installed iTerm2, zsh, and prezto. You’ll customize the look of the window and the tab behavior in the iTerm2 preferences and customize the text theme and plugins via prezto.

Since you’re configuring your plugins via prezto using files like the .preztorc, you shouldn’t need to customize zsh much directly.

Windows comes with two shells built in, the Command shell (cmd) and the PowerShell. The command shell is older and has been superseded by PowerShell. If you’re just getting started, you absolutely should just work with PowerShell. If you’ve been using Command shell on a Windows machine for a long time, most Command shell command work in PowerShell, so it may be worth switching over.

Once you’ve installed PowerShell, many people like customizing it with Oh My Posh.

7.2 Accessing Servers via SSH

Now that you’re set up with a terminal and shell, let’s dig into how to SSH into a remote server.

SSH – short for Secure (Socket) Shell – facilitates secure two-way communication between your computer and the command line on a server. Using SSH, you can provide a server with commands and rest easy that they’re going where you think and that the output is securely coming back.

When you’re interacting with a server using SSH, you’ll have a terminal open on your laptop, but all the commands are actually running off on the remote server. This means that if you are using a Windows machine, you’ll use Windows commands to open the SSH connection to the server and then switch to using Linux commands to actually administer the server.

Logging into a server is straightforward once its configured, you’ll just use ssh <server url> to get an interactive shell on that server.

Debugging SSH

SSH has one of my favorite debugging modes.

If you’re ever struggling with SSH, just add a -v to your command for verbose mode. If that’s not enough information, add another v for -vv, and even another!

Every v you add (up to 3) will make the output more verbose.

But in order to get there, you’ll need to configure ssh keys. SSH keys are what allows your computer and the server to mutually verify identity and set up a secure connection.

SSH keys come in pairs with two components – the public key and the private key.

I think these terms are somewhat of a misnomer. It would actually be better to call the private key the key and the public key the lock.

This is why you can give your public key to a server or service that you might not fully control. Someone who has your public key can verify that your private key is the one that fits that public key – but it’s basically impossible to reverse engineer the private key with the public key in hand.

Public key cryptography is cool

As an SSH user, you absolutely do not need to understand the underlying mathematics of public key cryptography, but conceptually quite simple. So forgive me this digression.

Public key cryptography relies on mathematical operations that are easy in one direction, but really hard to reverse.

Let’s say I ask you to find two numbers to multiply to equal 91 (\(1 * 91\) doesn’t count). It’ll probably take you at least a minute to discover that the only numbers that work are 7 and 13.

But if I were to give you 7 and 13 from the beginning, it’d be quick to check that \(7 * 13 = 91\).

That’s how public key cryptography works – your public key is a number with only two factors and your private key is those two factors. Logging in with the private key is quick, because checking multiplication is easy, but you can share the public key as broadly as you want, because reverse-engineering the two factors is hard. The difference is that actual public key cryptography doesn’t use small numbers like 91. It uses numbers with 91 or 9,191 digits.¹

SSH public keys are very big numbers and modern encryption standards mean that it’s basically impossible to break a public SSH key.

However, it is totally possible to compromise the security of an SSH connection by being sloppy with your private keys. So while SSH is cyptographically super secure, the whole system is only as secure as you. Always keep your private keys securely in the place where they were created and share only the public keys.

So that means if you need to log in to your server from your laptop, you’ll create a keypair on your laptop and share your public key to the server. If you need to SSH from the server to another server or to a service that uses SSH, like GitHub, you’ll create another SSH key on the server and use that public key on the far end of the connection.

7.3 Understanding bash commands

Now that we understand how to connect, let’s dig into what we’re going to be doing when we get there. Because the server we’ll be running is a Linux server, you’ll be using bash when you get there, regardless of the operating system on your computer.

The philosophy in bash is to provide small programs that each do one small thing well. The standard patterns for using bash commands involve the command itself, arguments, and options.

A command is the program you tell the command line to run, usually an abbreviation of the word for what you want to do. For example, the command to list the contents of a directory is ls.

Arguments tell the command what to run on. They come after the command with a space in between. For example, if I want to run ls on the directory /home/alex, I would run ls /home/alex on the command line.

Many commands have default arguments. For example, ls runs by default on the directory I’m currently in. So if I’m in /home/alex, running ls and running ls /home/alex would return the same thing.

Options or flags are sub-commands that modify how the command operates. Flags are denoted by having one or more dashes before them. For example, the ls command, which lists files, has the flag -l, which indicates that the files should be displayed as a list.

Flags always come in between the command and any arguments to the command. So, for example, if I want to get the files in /home/alex as a list, I can run ls -l /home/alex or navigate to /home/alex and run ls -l.

Some flags have arguments themselves. Arguments to flags appear after the flag with a space in between. So, for example, if you’re using the -l flag on ls, you can use the -D flag to format the datetime when the file was last updated.

So, for example, running ls -l -D %Y-%m-%dT%H:%M:%S /home/alex will list all the files in /home/alex with the date-time of the last update formatted in ISO 8601 format (which is always the correct format for dates).

It’s nice that this structure is standard. You always know that a bash command will be formatted as <command> <flags + flag args> <command args>. The downside is that having the main argument come all the way at the end, after all the flags, can make it really hard to mentally parse commands if you don’t know them super well. There’s no magic trick to make it easier, but you’re not alone – and there is help available!

All of the flags and arguments for commands can be found in the program’s man page (short for manual). You can access the man page for any command with man <command>. You can scroll the man page with arrow keys and exit with q.

Bash commands can get long, so it’s often nice to split them over multiple lines. You can tell bash you want it to keep reading after a line break by ending the line with a space and a \. Here’s what that might look like:

> cmd1 arg1 \
   arg2 \ 
   arg3

Symbol	What it is
man	manual
q	Exit man pages (and many other situations)
\	Continue command on new line

7.4 Basic navigation and interaction

Being on the command line is different from having a desktop with a mouse or touchpad to click on things. So the first thing to learn is how to navigate around a Linux environment and interact with the things you find there using Linux commands.

In Linux, directories define where you can be. A directory is just a container for files and other directories. In Linux, the entire file system is a tree (or perhaps an upside-down tree). The root directory, / is the base of the tree and a / in between two directories means that it’s a sub-directory. So the directory /home/alex is the alex directory, which is contained in home, which is in the root directory.

In Linux, every directory is a sub-directory of / or a sub-directory of a sub-directory of / or…you get the picture. The sequence of directories that defines a location is called a file path.

Every Linux command happens at a particular file path – called the working directory – and as a particular user. It won’t always matter, but in some cases the commands you’re allowed to run and what happens when you do so might be really different for one user vs another or the same user in different places.²

7.4.1 How does / compare to C:?

If you’re a Windows person, you might think this is analogous to C:. You’re not wrong, but the analogy is imprecise.

In Linux, everything is a sub-directory of /, irrespective of the configuration of physical or virtual drives that houses the storage. Frequently, people will put extra drives on their server – a process called mounting – and house them at /mnt (short for…you guessed it).

That’s different from Windows. In Windows you can have multiple roots, one for each physical or logical disk you’ve got. That’s why your machine may have a D: drive, or if you have network shares, those will often be on M: or N: or P:, each with its own sub-directory.

At any time, you can get the path where you’re sitting with the pwd command, which is an abbreviation for print working directory. When you’re writing out a file path, the working directory is at ..

File paths can be either absolute – that is specified relative to the root – or relative to the working directory. You can always tell the difference because an absolute directory starts with / and a relative one will start with something else, often ..

In Chapter 7, we talked about the ls command, which lists what’s in a directory. Now, you can understand that ls just has a default argument of ., so ls /home/alex will always return the contents list of /home/alex. If you’ve navigated there, ls and ls . would also return exactly the same thing.

Depending on what you’re doing, either absolute or relative paths make more sense. In general, absolute file paths make more sense when you want to access the same resource regardless of where the command is run, and relative file paths make more sense when you want to access a resource specific to where you run it.

Along with being able to inspect directories, it’s useful to be able to relocate there. You can change your working directory with the cd command, short for change directory.

Aside from / and ., there are two other special directories: .. and ~. .. is the parent of the directory you’re in, so you can move to the parent of your current directory using cd .. and to it’s parent with cd ../... ~ is the home directory of your user (assuming it has one). We’ll get more into what that means in a bit.

Recap of commands in this section

Command	What it does/is	Helpful options	Example
/	system root
.	current working directory
ls	list objects in a directory	-l - format as list -a - all (include hidden files)	$ ls . $ ls -la
pwd	prints working directory		$ pwd
cd	change directory		$ cd ~/ Documents
~	home directory of the current user		$ ls ~

7.4.2 Reading text files

Many of the files you’ll interact with as a Linux admin are just text files. In particular, application configuration files and logs are usually just text files.

A very common pattern in Linux administration is to read a log file to look for errors or clues, adjust a configuration setting as a result, and then restart a process.

You’ll find that your skills in understanding the Linux file tree, moving around, and seeing what’s in directories will be very helpful in getting to the files. Once you’re there, it’ll be useful to know how to actually interact with files.

The commands you’ll use most often will be cat and tail. cat is the command to print a file, starting at the beginning. It’s often helpful to read through text files. Sometimes you’ve got a really big file and you want to see just the beginning. less is useful if you just want to start at the top with the ability to scroll down. head is especially useful to peer at the beginning of a plain text data file (like csv) as it prints the first few rows and exits – so you can preview the beginning of a very large data file very quickly.

tail skips right to the end of a file. Log files usually are written so the newest part is last – so much so that “tailing a log file” is a synonym for looking at it. In some cases, you’ll want to tail a file as the process is still running and writing information to the log. You can get a live view of the end of the file using the -f flag (for follow).

Sometimes you want to search around inside a text file. You’re probably familiar with the power of regular expressions (regex) to search for specific character sequences in text strings. The Linux command to do regex searches is grep, which returns results that match the regex pattern you specify.

Using grep well requires being quite proficient in regex, so I usually just use it for simple searches.

The true power of grep is unlocked in combination with the pipe. The Linux pipe operator – | – takes the output of the previous command and sends it into the next one.

The pipe should feel extremely familiar to R users.

The {magrittr} pipe, %>%, has become extremely popular as part of the tidyverse since its introduction in 2013. A base R pipe, |>, was released as a part of R 4.1 in 2021.

sThe original pipe in {magrittr} took inspiration from both the Unix pipe and the pipe operator in the F# programming langauge.

So, for example, a combination I do all the time is to pipe the output of ls into grep when searching for a file inside a directory. So if I was searching for a file that contained the word data somewhere in the filename inside a specific project directory, that might look something like ls ~/projects/my-project | grep data.

Command	What it does	Notes + Helpful options
cat	Prints a file.
less	Prints a file, but just a little.	Can be very helpful to look at a few rows of csv. Lazily reads lines, so can be much faster than cat for big files.
tail	Look at the end of a file.	Useful for logs, where the newest part is last. The -f flag is useful for a live view.
head	Look at the beginning of a file.	Defaults to 10 lines, can specify a different number with -n <n>.
grep	Search a file using regex.	Writing regex can be a pain. I suggest testing expressions on regex101.com. Often useful in combination with the pipe.
|	the pipe

7.4.3 Managing files and directories

You can copy a file from one place to another using the cp command. cp leaves behind the old file and adds the new one at the specified location. You can move a file with the mv command, which does not leave the old file behind.

If you want to remove a file entirely, you can use the rm command. There is also a version to remove a directory, rmdir.

Warning

Be very careful with the rm command.

Unlike on your desktop there’s no recycle bin! Things that are deleted are instantly deleted forever.

If you want to make a directory, mkdir makes a directory at the specified filepath. mkdir will only work if it’s creating the entire file path specified, so mkdir dir/sub_dir will only work if neither dir nor dir/sub_dir works. Using mkdir -p dir/sub_dir, will make whichever parts of the path don’t yet exist.

If you want to operate on every file inside a directory, you can use the wildcard, *. So let’s say you’ve got a /data directory that has a bunch of CSV files in it. You could create a new data directory with mkdir new_data and copy all the files in using cp data/* new_data. You can also use the wildcard for partial matches, so if /data included a mixture of CSV and PDF files, and you only wanted to copy the CSVs, you could do cp data/*.csv new_data.

Command	What it does/is	Notes + Helpful Options	Example
rm	remove: delete permanently!	-r - recursively a directory and included files -f - force - dont ask for each file	$ r m - rf old_docs/ BE VERY CAREFUL WITH -rf
cp	copy
mv	move
*	wildcard
mkdir	make directory

7.4.4 Moving things to and from the server

It’s very common to have a file on your server you want to move to your desktop or vice versa. There are a few important commands to know to move things back and forth.

It’s generally easier to move a single file rather than a whole bunch. The tar command turns a set of files or whole directory into a single archive file, usually with the file suffix tar.gz. It also does file compression when it creates the archive file. The amount of compression depends on the content.

The tar command is used to both create and unpack (extract) archive files and telling it which one requires the use of several flags. I never remember them – this is a command I google every time I use it. The flags you’ll use most often are in the cheat sheet below.

Once you’ve created an archive file, you’ve got to move it. The scp command is the way to do this. scp – short for secure copy – is basically a combo of SSH and copy.³ Many ssh flags like -i to specify a particular SSH key apply to scp.

Since scp establishes an SSH connection, you need to make the request to somewhere that is accepting SSH connections. Hopefully your server is receiving SSH connections and your laptop is not. You’ll almost certainly have the experience at some point of being on your server and wanting to scp something to or from your laptop. You need to do the scp command from a regular terminal on your laptop, not one that’s already SSH-ed into your server.

Command	What it does	Notes + Helpful options
tar	compress/decompress file/directory	Almost always used with flags. Create is usually tar -czf <a rchive name> <file(s)> Extract is usually t ar -xfv <archive name>
scp	Copy across ssh	Can use most ssh flags

7.4.5 Writing files on the command line

There will be many situations where writing into a text file will be handy while administering your server – for example, when changing config files. When you’re on the command line, you’ll use a command line tool for writing into those files – meaning you’ll only have your keyboard to navigate, save, and exit.

There are times when you want to make files or directories with nothing in them – the touch command makes a blank file at the specified file path.

Command	What it does	Notes
touch	Creates file if doesn’t already exist.	Updates timestamp to current time if it does exist

7.4.5.1 Command line text editors

There are two command line text editors you’ll probably encounter – both extremely powerful text editing tools: nano and vi/vim.⁴

You can open a file in either by typing nano <filename> or vi <filename>. Unfortunately for many newbie Linux Admins it’s extremely easy to get stuck inside a file with no hint of how to get out!

In nano there will be helpful prompts along the bottom to tell you how to interact with the file, so you’ll see once you’re ready to go, you can exit with ^x. But what is ^? On most keyboards, you can insert the caret character, ^, by pressing Shift + 6. Pressing that key doesn’t seem to have any effect.

In this case, the ^ caret is short for Ctrl on Windows and for Command (⌘) on Mac. Phew!

Where nano gives you helpful – if obscure – hints, vim leaves you all on your own. It doesn’t even tell you you’re inside vim!

This is where many people get stuck and end up having to just exit and start a new terminal session. It’s not the end of the world if you do, but knowing a few vim commands can help you avoid that fate.

One of the most confusing things about vim is that you can’t edit the file when you first enter. That’s because vim keybindings were (1) developed before keyboards uniformly had arrow keys and (2) are designed to minimize how much your hands need to move.

When you enter, you’re in normal mode, which is for navigating through the file. If you feel like taking the time, learning vim keybindings can make navigating and editing text (code) files easier. I’d recommend spending some time learning! But if you don’t want to, you can skip all that by just pressing the i key.

Pressing the i key activates insert mode. For those of us who are comfortable in a word processor like Word or Google Docs, insert mode will feel very natural. You can type and words will appear and you can navigate with the arrow keys.

Once you’re done writing, you can go back to normal mode by pressing the escape key. In addition to navigating the file, normal mode allows you to do file operations like saving and quitting.

File operations are prefixed with a colon :. The two most common commands you’ll use are save (write) and quit. You can combine these together, so you can save and quit in one command using :wq.

Sometimes you may want to exit without saving. In the course of administering a server, this can also happen if you open a file you’re allowed to read, but not to write.

If you try to exit with :q, you’ll again find yourself trapped in an endless loop of warnings that your changes won’t be saved. You can tell vim you mean it with the exclamation mark and exit using :q!.

Vim Command	What it does	Notes + Helpful options
^	Prefix for file command in nano editor.	Its the ⌘ or Ctrl key, not the caret symbol.
i	Enter insert mode in vim
escape	Enter normal mode in vim.
:w	Write the current file in vim (from normal mode)	Can be combined to save and quit in one, :wq
:q	Quit vim (from normal mode)	:q! quit without saving

7.5 Lab: Accessing your server

If you followed the lab in Chapter 6, you set up an EC2 server. In this lab, we’ll SSH in so you’re ready to start administering the server in the next chapter.

7.5.1 SSH into the server

The .pem key you downloaded when you set up the server is the skeleton key – it will automatically let you in with complete admin privleges. In Chapter 8, we’ll set up a user with SSH on the server with more limited permissions.

In the meantime, we’re going to use the .pem key to get started on the server, but be extremely careful with the power of the .pem key.

Because the keypair is so powerful, AWS requires that you restrict the access pretty severely (more on what that means in Chapter 8). If you try to use the keypair without first changing the permissions, you’ll be unable to and get a warning that looks something like:

\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@

\@ WARNING: UNPROTECTED PRIVATE KEY FILE! \@

\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@\@

Permissions 0644 for 'do4ds-lab-key.pem' are too open.

It is required that your private key files are NOT accessible by others.

This private key will be ignored.

Load key "do4ds-lab-key.pem": bad permissions

ubuntu\@ec2-54-159-134-39.compute-1.amazonaws.com: Permission denied (publickey).

Before we can use it to open the server, we’ll need to make a quick change to the permissions on the key.

[QUESTION – does this happen on Windows? If so, what are commands to alter permissions?]

So let’s change the file permissions.

We’ll get into the details of how to use these commands in just a minute. For now, you’ll need to open a terminal window, navigate to the directory where the key is and change the file permissions.

On my machine that looks like:

$ cd ~/Documents/do4ds-lab
$ chmod 600 do4ds-lab-key.pem

You can sub in the path to where your key is and the name you used for your key.

In your terminal type the following

$ ssh -i do4ds-lab-key.pem ubuntu@$SERVER_ADDRESS

SSH on Windows

For a long time, Windows didn’t come with a built in SSH client, so you had to use PuTTY to do SSH from a Windows machine. Microsoft brought a native SSH client to Windows 10 in 2015, and it has been enabled by default since 2018.

If you run into any trouble using SSH commands on Windows, double check that you’ve enabled the OpenSSH Client.

Type yes when prompted, and you’re now logged in to your server!

7.6 Comprehension Questions

1. Draw a mental map that includes the following: terminal, shell, operating system, my laptop, server, ssh public key, ssh private key
2. If you don’t know the real commands for them, make up what you think the bash commands might be to do the following. So if you think you’d create a command called cmd with a flag -p and an argument arg, you’d write cmd -p <what p does> <arg>. In the next chapter you’ll get to see how close you got to the real thing:

   1. Change Directories, the only argument is where to go

   2. Making a Directory, with an optional flag to make parents as you go. The only argument is the directory to make.

   3. Remove files, with flags to do so recursively and to force it without checking in first. The only argument is the file or directory to remove.

---

1. It’s worth noting that modern encryption methods use substantially more convoluted mathematical operations than simple multiplication – but the idea is completely the same, and prime numbers are equally important.

2. It’s interesting to note that this is also true on your computer - when you open a program, a particular user is running a program on your computer and opening a GUI window for you to interact with. The point-and-clicking obfuscates this, but it’s still true.

3. It’s worth noting that scp is now considered “insecure and outdated”. The ways it is insecure are rather obscure and not terribly relevant for your workbench server. But if you’re moving a lot of data, you may want something faster. If so, I’d recommend more modern options like sftp and rsync. I probably wouldn’t bother if you’re not having issues with scp because these other commands have downsides like needing to learn an obscure syntax (sftp) or install an extra bit of software (rsync).

4. vi is the original fullscreen text editor for Linux. vim is its successor (vim stands for vi improved). For our purposes, they’re completely interchangeable.