Step-by-Step SSH Guide: From Basic Connection to Secure Configuration

June 11, 2025

When we talk about SSH, we're usually talking about the SSH Client, which is part of the OpenSSH suite of tools. Those tools are used to securely access remote machines, run commands on them, and handle incoming connections, among other things.

We need SSH, because most of us aren't working with servers located at home or at work. We need to be able to log into them over the network in a secure way. That's where SSH (secure shell) comes in.

This is a step-by-step walkthrough that goes beyond just the commands. It's detailed and intended for those that are seeking to better understand SSH. For this to go smoothly, it'd be best to add a fresh Ubuntu server that you can discard at the end.

What we're going to cover:

Key files and directories
Initial connection and password authentication
Creating a user account
Public key authentication setup
Configuring the SSH daemon
Local SSH configuration and shortcuts
Troubleshooting, solutions, and tips

What you'll need in order to follow along:

OpenSSH Client (likely already installed on your system)
A remote server running Ubuntu
The IP address of the server
The root user password of the server
Basic command line knowledge
Familiarity with a text editor (nano or vim)

Note: A pair of virtual machines would do as well, however, this guide doesn't include instructions on how to configure them.

The commands throughout were run on Ubuntu machines both locally and remote. If you're using Windows, then the WSL environment may work.

For those running macOS or a GNU/Linux distribution, OpenSSH Client is installed. According to Microsoft: "The latest builds of Windows 10 and Windows 11 include a built-in SSH server and client ... You can also check that it is present in Windows Settings > System > Optional features, then search for 'OpenSSH' in your added features." Ensure that OpenSSH Client is installed.

Now that we have all that out of the way, let's get started.

Open the terminal on your local machine. In your user's home folder, there may be a hidden .ssh directory. One way to confirm, is to try changing into it by running cd .ssh. If that works, then go ahead and list its contents with ls. You may see a file named authorized_keys. For our purposes, this file isn't important. However, on the server end, it's important, and we'll get to that soon. If you don't have an .ssh directory, then please reference the section on creating an .ssh directory.

Ensure that your working directory is your home one by running cd. We're going to log into that server now, and see what that looks like.

On a freshly provisioned server, unless an additional user was created during setup, the only available login account will be the root user.

To log in, we'll run ssh root@ followed by the server's IP address. For example, if the server's IP address is 172.233.178.14, we'll run ssh root@172.233.178.14.

Once we run that command, we'll be presented with a message similar to this:

The authenticity of host '172.233.178.14 (172.233.178.14)' can't be established. ED25519 key fingerprint is SHA256:saSL8FlKTMMWnCS8LWHOtKGIgNqhraHRNj7HyV0mEz8. This key is not known by any other names. Are you sure you want to continue connecting (yes/no/[fingerprint])?

We're seeing this message because we're logging in for the first time. If this were a server that we've accessed many times before, it would be a red flag. While unlikely, there would also be a risk that we're the target of a man-in-the-middle (MITM) attack. In that case, we'd need to confirm that we're actually connecting to the server we think we are.

Let's enter yes to the prompt. Next, we'll enter the root user password to complete authentication.

Now that we've successfully logged in, run clear to clean up the screen. We're on the server now, but something important has happened back on the local machine. Two new files have appeared in the .ssh directory. Those files are known_hosts and known_hosts.old. Let's stay on the server, but I'm going to explain what those files are.

The server's host keys (which are public keys) are written to the known_hosts file on the local machine. If these keys change, we'll receive the warning we saw a moment ago. This is a key integrity check, and it protects against man-in-the-middle attacks.

The known_hosts file records the host keys of any servers we connect to. For example, on this server, we can run: ls -1 /etc/ssh/*.pub. This lists all files in /etc/ssh ending with .pub, with one file per line. These keys are how the server proves its identity, and one of them is what the SSH client used to verify the server while we were connecting.

Alongside known_hosts on the local machine, a backup file named known_hosts.old was created. This is managed automatically and typically updated when ssh-keygen -R is executed. It retains the previous known_hosts state in case we need to restore it.

To view one of the server's host keys directly, we can run: cat /etc/ssh/ssh_host_ed25519_key.pub.

If we were to print the contents of the known_hosts file on the local machine, we'd find this key, along with possibly others from this or other servers we've connected to.

Our SSH client has an option called UpdateHostKeys that's enabled by default, so we're going to accept additional host keys from the server. This matters, because the additional host keys may prevent a warning prompt in the future if keys are rotated. The reason I'm telling you, though, is so you'll know why there's more than one key in the local known_hosts file even though we've connected to one server. This is how that played out: Initially, the server responded with the primary host key. We didn't have a matching key, or a known_hosts file for that matter, and we proceeded with the connection when we entered "yes". The host key was then written to our newly created known_hosts file. Then, we submitted the root user password and completed authentication. The server then responded with the remaining available keys, and they were written to our known_hosts file on the local machine.

Let's return to the local machine by running exit.

This first time around, we logged in using a password. Unless there's a reason that password authentication must be used, then we shouldn't use it. The trouble with password authentication is that passwords can be defeated using brute force. We could allow passwords, and then automatically ban someone if they make too many attempts to reduce the effectiveness of brute-force attacks.

There's another form of authentication that we can use though, and that's public key authentication. This form of authentication involves a key pair. There's no password to guess. Instead, a pair of keys are used to access the server. We can generate a key pair by running a command that will provide us with a public key and a private key. For what we're doing, a copy of the public key goes on the server. The private key, on the other hand, stays on the local machine, and we should protect it through a combination of permissions and encryption. We're not concerned with protecting the public key.

During public key authentication, the client responds to the server with a signature that was created using the private key. That signature is validated by the server using the corresponding public key. If the signature is valid, then authentication is successful. This is why we should encrypt the private key using a passphrase. If someone else controls your private key, then they can sign as if they are you. They can access a system as if they are you. I use passphrases, because I administer servers other than my own, and the stakes are higher. Anyone accessing important systems should consider using passphrases. If we choose to use a passphrase when creating a key pair, the private key is encrypted automatically.

So what's the command to create a key pair?

Key pairs are created by running the ssh-keygen command.

On the local machine, let's run ssh-keygen -f ~/.ssh/mykey "mykey" could be something else. It could be the hostname of the server if we know what that's going to be. Option -f allows us to specify the name of the key pair and where to create it. If we'd left this out, we would've been prompted to provide it.

We'll then see some output that reads:

Generating public/private ed25519 key pair.
Enter passphrase (empty for no passphrase):

A recent version will default to the ed25519 cryptographic algorithm. If we had a need to use a different type, we could specify that by using option -t. If we were doing that, then we might have looked into option -b for bits.

For the passphrase, if it's a key that you're going to continue to use after completing this, then I suggest entering a strong one. The key pair will then be saved to the .ssh directory in the home folder. This can be verified by running ls ~/.ssh. When you create keys for your servers, back them up.

We need to get the public key onto our server, so that we can log in using this key pair. The contents of the public key belong on the server in a file called authorized_keys within our user's .ssh directory.

At the moment, we only have the root user on the server. We should create our own user account instead of logging in as root directly. We're going to disable root logins for security, so we'll need our own user account to access the server. Our user can still have administrative privileges, but it won't be the root account itself. Once we create this user account, we'll add our public key to its authorized_keys file and use it for future logins.

Let's start by accessing the system as root like we did before, so we can get that account created.

Run ssh root@server-ip. In my case, ssh root@172.233.178.14.

Then, enter the root password.

Create a user with the useradd command: useradd -m -s /bin/bash username. For example, useradd -m -s /bin/bash charles.

Option -m creates a home directory for the user, and option -s sets the path to the user's login shell.

I'm using my name Charles here, but it's not a good idea to use your name. I've seen attempts to access my server using my name. I don't use my name though, so it was obviously a bot or someone trying my name.

Then, set a password with the passwd command followed by your username: passwd charles.

You'll be prompted twice for the password.

We're going to need an account with superuser privileges, so let's add our user to group sudo. This will allow us to run any command as any user. We can add sudo to the list of groups the user belongs to like this: usermod -aG sudo charles.

This is like saying modify user charles by appending group sudo to the list of supplementary groups.

We haven't disabled password authentication yet, so at this point we could log in as our user by using a password. Let's run exit on the server to exit out and return to the local machine. Next, we're going to get that public key up to our user's .ssh directory on the server. There's a command that we can use called ssh-copy-id. We can copy the key up like this:

ssh-copy-id -i ~/.ssh/mykey charles@172.233.178.14

You'll then have to provide your user password to add the key.

We're using option -i there, because we want to copy a particular public key, in this case, mykey.pub to the server. If we leave off the .pub it'll be added automatically. In the process, a login attempt is made to determine if the public key is already present in the authorized_keys file on the server. If it's not, then it's copied up and appended to that file. If authorized_keys doesn't exist, or even if the .ssh directory doesn't exist, then the .ssh directory and authorized_keys file will be created automatically for us.

Now that the key has been added, log in as we have been, but with the new account username. If your key pair was created using a passphrase, then you'll be prompted for it. If you're running Ubuntu on your local machine, then below the field where you enter the passphrase, there's a checkbox that reads something like "Automatically unlock this key whenever I'm logged in". When you check that box, your decrypted private key will be held in memory by the ssh-agent, so that you don't have to enter it every time. This is convenient, however, it may be best to restrict this to your personal computer. Even then, if somebody has access to your unlocked computer, that could negate the benefits of a passphrase. Consider locking your laptop when you step away if you're using it around others. Once the passphrase has been entered, you'll be logged in as usual. If you weren't presented with the option to add the key to the authentication agent, and you want to add the key to the agent, then check out ssh-add. You can learn more by running man ssh-add. Note that keys added to ssh-agent don't persist across reboots by default.

We're using public key authentication now, but the server still accepts password authentication. We should disable that. We can also still log in as root directly, which is a security risk. If we need root access, it's better to log in as a user with superuser privileges first, then switch to root from inside the system. Let's configure the SSH daemon to turn off these options and make some other common adjustments.

If you're no longer connected to the remote server, then log back in as your new user.

Before we start configuring the SSH daemon, it's helpful to understand what we're working with on the server. There's an openssh-client package, and an openssh-server package. Commands such as ssh, ssh-keygen, and ssh-copy-id are provided by the openssh-client package. The openssh-server package provides the sshd server and its configuration files. sshd is the daemon that listens for incoming SSH connections. The ssh command is used to initiate a connection to a remote server.

The folders and files that we're going to work with are owned by root. If you haven't already, add your user to the sudo group.

Let's run ls /etc/ssh to see what files are present in the ssh directory. One of those files is sshd_config. One way we can make these configuration changes is to modify that file, and then restart the service. That's fine, but at times, a new version of that file will become available, and we'll be asked if we want the new one or to keep the existing one as we're upgrading packages. We're going to want the new one, but we're not going to want to lose our changes. When this happens, we have the opportunity to compare the differences, but there's an easier way to go about this. In /etc/ssh, there's a directory named sshd_config.d. We can make our changes in that directory, and they'll override the settings in sshd_config. When a new version of the sshd_config file becomes available, we can just accept the new one without worrying about losing our changes.

Change directories to /etc/ssh/sshd_config.d with cd /etc/ssh/sshd_config.d. Then, run sudo touch overrides.conf. Enter your password to finish creating the file.

Before we add our configuration settings to this file, let's check out the sshd_config file. Run less ../sshd_config. As you can see there are lots of settings in here. Some are commented and some are not. Commented ones are defaults, but if the same setting is present and it's not commented, then the uncommented setting will override the commented one.

#PasswordAuthentication yes
PasswordAuthentication no

You can scroll down by using the down arrow or by paging down. When you're done looking, just hit q to exit. I use a combination of these settings that make sense for my use case. You can learn more about each of these settings anytime by reading the man page for the SSH daemon configuration file. To do that run man sshd_config.

Now we're going to add our overrides. Our overrides.conf file is owned by root, so we'll have to edit it as root. Using your preferred editor (typically vim or nano), edit the file as root.

sudo nano overrides.conf

sudo vim overrides.conf

Add these three to the file and replace charles with your user.

PermitRootLogin no
PasswordAuthentication no
AllowUsers charles

So, we're not allowing password authentication or root logins. Even if PermitRootLogin were to be reverted somehow, users can't log in using a password. As long as root doesn't have a key pair, someone can't log in as root anyway. On top of that, we're restricting allowed users to ourselves. These are a few common and important settings. It's also worth mentioning the Port setting. Changing the port can be helpful in the sense that your logs may be less polluted, and maybe you'll avoid some of the less sophisticated bots that are targeting port 22. Understand though, that changing the port doesn't stop an attacker that is targeting your server directly.

A couple of things you can do that go a long way are:

Keep your packages up-to-date.
Review your SSH daemon configuration periodically to check for misconfigurations and disable unused features to minimize your attack surface.

If you do want to use a different port, then check out the instructions I've provided in the changing the SSH port section.

Save and exit the file. Whenever we make a configuration change to the SSH daemon, we have to restart the service in order for the changes to take effect. Before we restart the service, we can check the validity of the configuration by running sudo sshd -t. If there's no output, then it's good to go. Restart the service by running sudo systemctl restart ssh. Take a look at the current status of the service by running sudo systemctl status ssh. On the Active line, you'll see at the end of the line how long ago the service last started.

Active: active (running) since Sat 2025-06-07 05:16:14 EDT; 5s ago

We configured the SSH daemon, so it seems strange that we restarted SSH and not the SSH daemon (ssh.service as opposed to sshd.service). In that same status output, there's a line that says Loaded, and that line includes a path to the SSH unit file (ssh.service). If we print the contents of that file by running cat /usr/lib/systemd/system/ssh.service, we'll see Alias=sshd.service at the bottom. An alias is set, so the sshd binary will be executed whether we run sudo systemctl restart ssh or sudo systemctl restart sshd.

Now that we've made these changes, and restarted the service, we can exit to our local machine by running exit. Go ahead and try logging into the remote machine as root now. You'll see output similar to

root@172.233.178.14: Permission denied (publickey).

because root doesn't have a public key on the remote machine. Even if there was one though, the connection wouldn't succeed due to our PermitRootLogin no setting. You could try enabling password authentication while leaving root logins disabled to see what that looks like. Just remember to restart the service. Otherwise, the changes won't have any effect.

We've covered the essentials, but I encourage you to check out the additional information below. I'm including bonus information like tips, troubleshooting, and other things that are useful to know.

By the way, if this was helpful to you and you'd like to show your appreciation, feel free to buy me a coffee. Every little bit helps, and your support means a lot! Thanks!

Bonus:

Using SSH to Execute Commands on a Remote Machine

If you just want to know how much memory is available on the server, you can include the command like this:

ssh charles@172.233.178.14 free -h

This will output information about the server's memory and you'll remain on the local machine.

Simplifying Logins with a Local Configuration File

If you've provided a custom name for your public key, then without a configuration file, your command will look like this:

ssh -i ~/.ssh/mykey charles@172.233.178.14

unless you used a passphrase and added your private key to the agent. Instead, you could run something like this:

ssh server

The name of the configuration file is config and it belongs in your .ssh directory on the local machine. Here's an example of what the contents look like:

Host server
   HostName 172.233.178.14
   User charles
   IdentitiesOnly yes
   IdentityFile ~/.ssh/mykey

Now, instead of including all those details in your command, you're storing them in your config file. That first line Host server is essentially the label you're giving the server. It could be the hostname of the server, or something else that you'll remember. Now, you can just run ssh followed by the host like so: ssh server. Use chmod 600 ~/.ssh/config to set the permissions on the config file to 600.

Changing the SSH Port

OpenSSH on Ubuntu is configured by default to use systemd socket activation.

On Ubuntu 24.04 LTS, the port and address settings are pulled dynamically from sshd_config via an executable called sshd-socket-generator, and a temporary file is created at:

/run/systemd/generator/ssh.socket.d/addresses.conf

Change the port in sshd_config or an override in sshd_config.d
Run sudo systemctl daemon-reload
Run sudo systemctl restart ssh.socket
Allow that port in the firewall configuration
If using an SSH config file locally, then update that host in the config file to include the new port like so Port 8522 or whatever the port is now.

Identifying Weak Algorithms Enabled on the System

Visit https://sshcheck.com and enter the server's hostname or IP address to quickly identify any weak algorithms that should be disabled.

Reviewing Detailed Authentication Information

These details can be found in /var/log/auth.log. Programs like Fail2ban monitor this log file.

Creating an .ssh Directory

If the .ssh directory is missing, create it from within your home folder with mkdir .ssh.

Check the directory permissions with ls -ld .ssh. You'll likely see something like drwxr-xr-x, but only you should have read, write and execute permissions on .ssh, because private keys are going to be created in it that must be protected. Those keys will be used to access servers. While your home folder permissions should prevent others from accessing it anyway, it's good practice to restrict .ssh permissions to yourself: chmod 700 .ssh.

Use ls -ld .ssh to confirm ownership and permissions. You should see something like:

drwx------ 2 charles charles 4096 Jun 07 05:37 .ssh

SSH Troubleshooting

Increase the verbosity:

Begin by using option -v for verbose output. This will provide you with detailed information about what's happening during the connection attempt.

ssh -v server

You can use up to three -v options to increase the verbosity.

ssh -vvv server

Doing this from the start may save you the time of checking logs, files, and other things.

Check the logs:

Look for messages like:

Could not open user 'charles' authorized keys '/home/charles/.ssh/authorized_keys': Permission denied

Where to check:

/var/log/auth.log

sudo journalctl -u ssh

Verify credentials and permissions:

Credentials

These are a few common issues to rule out.

Incorrect username or password/passphrase
Wrong key or mismatched key pair
Missing authorized_keys file or public key not added to it

Permissions

On the remote system, .ssh should be owned by the authenticating user, and permissions should be set to 700. The authorized_keys file should be owned by the authenticating user, and permissions should be set to 600. If either is not at least readable by the user, authentication will fail.

On the local system, .ssh should be owned by the authenticating user, and permissions should be set to 700. The private key should also be owned by the authenticating user, and have permissions of 600. While .ssh/config doesn't store secrets, it controls SSH behavior. Loose permissions could compromise its integrity and allow unexpected or malicious connection behavior. The pattern continues: the user should own their config file, and permissions should be set to 600. If the user cannot access their .ssh directory, read their private key, or read their config file, then authentication will fail.

Check network connectivity:

Port

The server may be listening on an unexpected port. To reach your system, you may have to use your cloud provider's console access. From your local machine, you could use a networking tool like nmap to identify open ports on your server. Then, if you find what port the SSH daemon is listening on, you can attempt to connect again. Otherwise, the console that's provided to you by your cloud provider will allow you to access your server.

If you access via the console, then one method to determine what port the SSH daemon is listening on is to run sudo ss -lp | grep sshd. The output may include something like *:8522. The portion after the colon is the port number. If you have a local config file, then make sure that's the port that's set. Otherwise, when you run ssh include the port as an argument to option -p. For example, ssh -p 8522 -i ~/.ssh/mykey charles@172.233.178.14.

This type of problem may occur if you configure the SSH daemon to listen on a different port. If you miss a step like restarting the service, then you may still be on port 22. I detail these steps above in the changing the SSH port section.

Firewall

The SSH port may not be allowed by the firewall. Port 22 is typically enabled by default by cloud providers. If it's not, or if you're using a non-standard port, then you'll have to allow that traffic through the firewall. You can accomplish that either by using the provider's GUI or by using a program like UFW.

Confirm that the service is running:

It's not likely the problem, but it's possible. If a socket like ssh.socket isn't configured to trigger the SSH daemon, then ensure that the SSH service is both running and enabled. You can check with sudo systemctl status ssh. The output should show that the service is running and enabled. If you see disabled on the Loaded line:

Loaded: loaded (/usr/lib/systemd/system/ssh.service; disabled; preset: enabled)

then run sudo systemctl enable ssh. If the service isn't running, then attempt to start it with sudo systemctl start ssh. The command sudo systemctl enable --now ssh both enables and starts the service. The service needs to be enabled, because there's no socket configured. If the system were to reboot, then the service wouldn't start. Confirm that the service is running: sudo systemctl status ssh. If it's not, then there's some kind of issue. Look towards the bottom of that status output for any errors like a syntax error.

Check configuration syntax:

It's a good habit to check for syntax errors after modifying configuration files. For the SSH daemon, you can run sudo sshd -t. No output means the syntax is correct. Nginx and Apache both have a similar tool. Making use of these syntax checkers can save you from causing a crash.

Thanks for reading! If this post was helpful, or if you have any questions, please comment below.