Case study: VPS 184.168.122.73 • Host ip-184-168-122-73 • Login user dsbokteam • Recovery work performed as root • Incident observed 3 November 2026 (server UTC).
Audience: Linux administrators managing GoDaddy VPS servers and other Ubuntu servers.
Result: ssh.service successfully became active (running) and enabled under systemd. A controlled post-reboot check later showed active / enabled, but the transcript does not explicitly establish whether a reboot occurred immediately before that check.
Scope and evidence: The SSH incident, exact errors, investigated services, and final working commands are reconstructed from the supplied console logs. The original 20.04 → 22.04 upgrade command transcript was not supplied; the pre-upgrade and release-upgrade procedure below is a recommended repeatable workflow, not a claim about commands executed in this incident. Do not run an upgrade again on a server already on 22.04.
1. Executive summary
The Ubuntu upgrade was followed by a loss of remote SSH access:
SSH Connecting to 184.168.122.73
Connection refused (os error 61)
Initially no process listened on TCP port 22. A manual attempt to check the SSH configuration reported:
Missing privilege separation directory: /run/sshd
Creating /run/sshd and launching /usr/sbin/sshd restored temporary access but not automatic startup. ssh.service was enabled but inactive; systemctl start ssh.service hung and timed out with code 124.
The underlying culprit was a custom /etc/systemd/system/sshd-host-keys.service with Before=ssh.service that executed dpkg-reconfigure --frontend=noninteractive openssh-server. The package’s post-installation script tried to systemctl restart ssh.service, which was still waiting for the custom service to finish: a circular wait. After the custom service was disabled and its stuck job cleared, another problem appeared: the manually launched sshd occupied port 22, causing systemd’s sshd to fail with Address already in use. The manual listener was stopped safely, and systemd started the managed daemon successfully.
2. Environment and recovery prerequisites
| Field | Incident value |
|---|---|
| Provider | GoDaddy VPS Hosting |
| Public IP | 184.168.122.73 |
| OS change | Ubuntu 20.04 LTS → Ubuntu 22.04 LTS |
| Normal SSH user | dsbokteam |
| Privileged shell | sudo -s to root |
| Systemd SSH unit | ssh.service |
| Problematic extra unit | /etc/systemd/system/sshd-host-keys.service |
| SSH port | TCP 22 |
| Working managed SSH PID in final incident log | 7275 (will change after restart) |
Safety prerequisites before changing remote access: Maintain an independent GoDaddy Recovery Console session; verify the VPS has a current restore point/backup; do not remove SSH host keys; do not flush firewalls; do not reboot while SSH cannot start under systemd. A shell over SSH is not a substitute for out-of-band recovery access.
GoDaddy Recovery Console versus Rescue Mode
- Sign in to GoDaddy → My Products / Servers → Manage the VPS.
- Choose Server Actions → Recovery Console.
- Log in with a valid Linux server account and password (not the GoDaddy web-account password). The working user in this incident was
dsbokteamand the root shell was obtained withsudo -s. - Rescue Mode is different: it boots into a rescue environment and GoDaddy provides temporary credentials. Use only if normal boot/console recovery is insufficient; it interrupts hosted services.
Provider references: https://www.godaddy.com/en/help/use-the-recovery-console-for-vps-hosting-32064 and https://www.godaddy.com/en/help/boot-my-vps-hosting-in-rescue-mode-32175
3. Recommended pre-upgrade checklist (reference procedure; not verified incident history)
Do this before a future Ubuntu release upgrade, in a scheduled maintenance window.
# Record current OS, kernel, system state and disk availability.
cat /etc/os-release
uname -a
df -h
free -h
systemctl --failed --no-pager
ss -tlnp
# Inspect custom services and SSH startup configuration.
systemctl cat ssh.service
systemctl list-unit-files '*ssh*'
ls -la /etc/systemd/system/*ssh* 2>/dev/null
sudo /usr/sbin/sshd -t
# Record installed packages and important configuration.
sudo install -d -m 0700 /root/pre-upgrade-records
sudo dpkg-query -W > /root/pre-upgrade-records/packages.txt
sudo cp -a /etc/ssh /root/pre-upgrade-records/ssh-config-backup
sudo cp -a /etc/apt /root/pre-upgrade-records/apt-config-backup
Security: /etc/ssh may contain private host keys. Keep that backup restricted to root, preferably encrypted/off-server, and do not publish it. In addition, take a provider snapshot or separately verified full backup of application data, databases, configurations and volumes. Verify recovery access before changing the OS.
Update the current Ubuntu release fully and install the official upgrader (run only while still on the old release):
sudo apt update
sudo apt dist-upgrade -o APT::Get::Always-Include-Phased-Updates=true
sudo apt install update-manager-core
sudo grep '^Prompt=' /etc/update-manager/release-upgrades
For LTS-to-LTS upgrades, Prompt=lts is appropriate. Read package prompts carefully, especially if /etc/ssh/sshd_config or third-party repositories have local customizations. Do not blindly overwrite an existing SSH configuration.
4. Recommended Ubuntu 20.04 → 22.04 upgrade procedure (reference only)
The official server upgrade mechanism is:
sudo do-release-upgrade
Run only on the pre-upgrade server with backups and recovery access ready. Read the proposed package removals/replacements before proceeding. Ubuntu’s release upgrader handles configuration questions interactively and typically runs in a recoverable GNU screen session for remote upgrades. Do not use -d for a routine production LTS upgrade.
At the end of a successful upgrade, a reboot is normally required. The release upgrade is not complete until that reboot. See Ubuntu’s official instructions: https://ubuntu.com/server/docs/how-to/software/upgrade-your-release/
Before rebooting, check for unexpected custom units such as sshd-host-keys.service. On this VPS it was a custom service outside Ubuntu’s regular ssh.service implementation and blocked boot-time startup.
5. Incident chronology: actual observed errors and meaning
| Stage | Exact observation | Technical meaning |
|---|---|---|
| SSH unavailable | Connection refused (os error 61) | TCP connection refused before authentication; not a bad password |
| Listener check | `ss -tlnp | grep ‘:22’` returned nothing |
| SSH validation | Missing privilege separation directory: /run/sshd | Required runtime directory was absent |
| Manual fallback | /usr/sbin/sshd worked | SSH binary could run outside systemd |
| Enable status | enabled / inactive | Configured for boot but not actually running as a managed service |
| Managed startup | timeout 20s systemctl start ssh.service → Exit code: 124 | Startup waited beyond 20 seconds |
| Systemd queue | sshd-host-keys.service start running and ssh.service start waiting | An earlier ordering/dependency blocker was active |
| Custom unit | Before=ssh.service; ExecStart=dpkg-reconfigure --frontend=noninteractive openssh-server | Custom host-key job reconfigured the package at boot |
| Reconfiguration child | deb-systemd-invoke restart rescue-ssh.target ssh.service | Host-key job tried to restart the service it preceded |
| After unblocking | Bind to port 22 ... Address already in use; fatal: Cannot bind any address. | Manual sshd had become the remaining port conflict |
| Final result | ssh.service: active (running), enabled, systemd-managed PID 7275 | Managed SSH startup succeeded |
Root cause: ordering and restart deadlock
systemd starts sshd-host-keys.service
|
v
ExecStart: dpkg-reconfigure openssh-server
|
v
openssh-server.postinst calls systemctl restart ssh.service
|
v
ssh.service is WAITING for sshd-host-keys.service
|
+------ circular wait / stalled boot transaction ------+
The file /etc/systemd/system/sshd-host-keys.service contained:
[Unit]
Description=Generates ssh host keys at boot time
Documentation=man:sshd(5)
Before=ssh.service
[Service]
Type=oneshot
RemainAfterExit=yes
User=root
ExecStart=dpkg-reconfigure --frontend=noninteractive openssh-server
[Install]
WantedBy=multi-user.target
This unit should not be assumed to exist on every Ubuntu server; it was specific to this VPS. In the incident, SSH host keys already existed (/etc/ssh/ssh_host_rsa_key, ssh_host_ecdsa_key, ssh_host_ed25519_key). No regeneration was necessary.
6. First temporary recovery: repair /run/sshd
From GoDaddy Recovery Console, authenticate as the Linux user and get root privileges:
sudo -s
Investigate the server:
cat /etc/os-release
ps -p 1 -o pid,comm,args
ps aux | grep '[s]shd'
ss -tlnp | grep ':22'
dpkg -l openssh-server
/usr/sbin/sshd -t
When sshd -t reports a missing runtime directory:
install -d -m 0755 -o root -g root /run/sshd
/usr/sbin/sshd -t
echo $?
# Expected: 0
The temporary emergency workaround that succeeded was:
/usr/sbin/sshd
ss -tlnp | grep ':22'
This is not a permanent boot fix. /usr/sbin/sshd starts a daemon without making systemd own the process. Once systemd tries to start its own instance, the manually created listener can conflict on port 22.
7. Investigate systemd rather than repeating a hanging command
# Show the service without letting output enter a pager.
systemctl status ssh.service --no-pager -l
systemctl is-enabled ssh.service
systemctl is-active ssh.service
# Bound startup attempts so the console does not hang indefinitely.
timeout 20s systemctl start ssh.service
echo "Exit code: $?"
# Identify the queued job or dependency.
systemctl list-jobs --no-pager
journalctl -u ssh.service -b -n 50 --no-pager
systemctl cat ssh.service
A timeout exit code of 124 means the command timed out; it does not by itself prove sshd has a corrupt configuration. In this case, systemctl list-jobs exposed the real blocker:
183 sshd-host-keys.service start running
120 ssh.service start waiting
175 systemd-update-utmp-runlevel.service start waiting
2 multi-user.target start waiting
1 graphical.target start waiting
Inspect that specific unit:
systemctl status sshd-host-keys.service --no-pager -l
systemctl cat sshd-host-keys.service
journalctl -u sshd-host-keys.service -b -n 50 --no-pager
ps -ef | grep -E 'dpkg-reconfigure|openssh-server.postinst|systemctl.*ssh'
ls -l /etc/ssh/ssh_host_*_key
The process tree showed dpkg-reconfigure, openssh-server.postinst, deb-systemd-invoke restart ... ssh.service and systemctl ... restart ssh.service. That confirmed the startup cycle.
8. Permanent repair, part A: remove the harmful boot-time dependency
Apply this only to the confirmed custom unit above, not to Ubuntu’s normal ssh.service. Keep GoDaddy Recovery Console open.
# Prevent the custom unit from being started during future boots.
systemctl disable sshd-host-keys.service
# Inspect the active job, if any, before stopping it.
systemctl status sshd-host-keys.service --no-pager -l
systemctl list-jobs --no-pager
In this incident the custom unit was stuck inside its own package reconfiguration. The recovery procedure was to stop that specific custom unit and, if it remained stuck, terminate only the processes in its cgroup:
# If stuck, terminate only the custom service's child processes.
systemctl kill --kill-who=all --signal=SIGTERM sshd-host-keys.service
systemctl stop sshd-host-keys.service
systemctl daemon-reload
Do not forcibly kill unrelated package upgrades. Confirm package manager jobs are not actively performing a different upgrade before intervening. If systemctl stop hangs, inspect jobs again; preserve the console connection.
Check the result:
systemctl is-enabled sshd-host-keys.service
# Observed final result: disabled
systemctl list-jobs --no-pager
# Observed final result: No jobs running.
Optional: If the unit repeatedly gets reactivated by another component, investigate that dependency. systemctl mask sshd-host-keys.service can prevent starts, but was not required for the successful final state recorded here. Be cautious with provider-customized units and document any local changes.
9. Permanent repair, part B: move SSH from a manual process to systemd
Once the dependency cycle was cleared, the managed start failed with:
error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
error: Bind to port 22 on :: failed: Address already in use.
fatal: Cannot bind any address.
This was because the manually started SSH listener owned TCP port 22. A later check also showed /run/sshd missing again. That can happen because ssh.service declares RuntimeDirectory=sshd and systemd ordinarily removes the runtime directory when the service stops; it is not proof that the filesystem is corrupted.
Exact working recovery sequence (run from independent GoDaddy console):
# 1. Recreate the runtime directory and validate configuration.
install -d -m 0755 -o root -g root /run/sshd
/usr/sbin/sshd -t
echo $?
# Observed: 0
# 2. Verify the custom job is disabled and no queued jobs remain.
systemctl is-enabled sshd-host-keys.service
systemctl list-jobs --no-pager
# Observed: disabled; No jobs running.
# 3. Identify the *current* listener. Do not assume an old PID is still valid.
ss -tlnp | grep ':22'
ps -p 6534 -o pid,ppid,args
# In the incident, PID 6534 was the standalone sshd listener.
# 4. Terminate ONLY the verified standalone listener, from Recovery Console.
kill -TERM 6534
# 5. Clear systemd's earlier failed state and start managed SSH.
systemctl reset-failed ssh.service
timeout 20s systemctl start ssh.service
echo "Exit code: $?"
# Observed: Exit code: 0
# 6. Verify startup and boot enablement.
systemctl status ssh.service --no-pager -l
systemctl is-enabled ssh.service
systemctl is-active ssh.service
# Observed: enabled; active
Important: 6534 was a real incident PID, not a reusable constant. For a different server or later boot, discover and verify the current listener before sending a signal. If your only access is SSH, do not kill its listener without an independent Recovery Console session. The final managed SSH process was PID 7275 and logged:
Started OpenBSD Secure Shell server.
Server listening on 0.0.0.0 port 22.
Server listening on :: port 22.
10. Why /run/sshd vanished and how it is managed at boot
Ubuntu’s ssh.service unit on this VPS includes:
ExecStartPre=/usr/sbin/sshd -t
ExecStart=/usr/sbin/sshd -D $SSHD_OPTS
Type=notify
RuntimeDirectory=sshd
RuntimeDirectoryMode=0755
Restart=on-failure
Normal behavior: When systemd starts ssh.service, it creates /run/sshd. When that service stops, systemd may remove the directory. The directory is temporary and should not be made persistent by storing it under a different filesystem.
During troubleshooting, this additional tmpfiles rule was created:
cat > /etc/tmpfiles.d/sshd.conf <<'RULE'
d /run/sshd 0755 root root -
RULE
systemd-tmpfiles --create /etc/tmpfiles.d/sshd.conf
This is a permissible additional boot-time safeguard, but not the main fix: correct ssh.service startup already creates /run/sshd automatically. The main fix was clearing the custom service deadlock and moving SSH under systemd management.
Reference: https://www.freedesktop.org/software/systemd/man/256/systemd.exec.html
11. Final validation and controlled reboot
Before rebooting, verify all of these:
# Should show active / enabled.
systemctl is-active ssh.service
systemctl is-enabled ssh.service
# Should show disabled (or masked if deliberately masked).
systemctl is-enabled sshd-host-keys.service
# Should show no blocked jobs.
systemctl list-jobs --no-pager
# Should show the managed daemon listening on 22.
systemctl show ssh.service -p ActiveState -p SubState -p MainPID -p RuntimeDirectory
ss -tlnp | grep ':22'
# Should validate cleanly.
/usr/sbin/sshd -t
Additional overall checks:
systemctl --failed --no-pager
dpkg --audit
df -h
From a new terminal on your Mac (use your established SSH authentication method):
ssh dsbokteam@184.168.122.73
Only after a fresh connection succeeds, your backup/recovery console is available, and ssh.service is active/enabled, conduct a controlled reboot during a maintenance window:
sudo reboot
After the server returns:
systemctl is-active ssh
systemctl is-enabled ssh
systemctl is-enabled sshd-host-keys.service
systemctl list-jobs --no-pager
ss -tlnp | grep ':22'
Observed at end of user log: systemctl is-active ssh → active; systemctl is-enabled ssh → enabled. The log does not explicitly say whether this check followed a reboot, so do not treat reboot persistence as independently proved until you verify it after a known reboot.
12. Additional upgrade health checks and only-if-needed repairs
After a major OS upgrade, assess applications independently from SSH:
cat /etc/os-release
uname -r
sudo systemctl --failed --no-pager
sudo dpkg --audit
sudo journalctl -b -p err --no-pager -n 100
sudo ss -tlnp
sudo df -h
Check only services actually installed on the server, for example nginx, apache2, mysql, mariadb, php*-fpm, firewalls, or monitoring agents. Third-party repositories may have been disabled during the distribution upgrade; inspect them before re-enabling compatibility-sensitive packages.
If dpkg --audit reports half-configured packages and no other apt/dpkg/release-upgrader job is running, the general Ubuntu repair tools are:
sudo dpkg --configure -a
sudo apt --fix-broken install
However: Because openssh-server reconfiguration specifically triggered the deadlock in this case, do not run those package-repair commands blindly while a problematic sshd-host-keys.service is still running or while SSH recovery is incomplete. Inspect any proposed changes and keep the Recovery Console available.
Troubleshooting matrix
| Symptom | Check | Correct response |
|---|---|---|
SSH Connection refused | ss -tlnp | grep ':22' | Check service/listener and port, not password first |
sshd -t says missing /run/sshd | ls -ld /run/sshd | install -d -m 0755 -o root -g root /run/sshd; then sshd -t |
systemctl start ssh hangs | systemctl list-jobs | Find waiting unit; inspect custom dependencies |
ssh.service enabled but inactive | systemctl status ssh | Enabled is only boot configuration; fix startup and retest |
sshd-host-keys.service stuck | systemctl cat and status | Identify dpkg-reconfigure / recursive SSH restart; disable confirmed custom unit |
Address already in use on port 22 | ss -tlnp | grep ':22' | Transition from verified standalone listener to systemd with out-of-band access |
ssh.service failed after a former error | journalctl -u ssh -b | Fix root problem, systemctl reset-failed ssh, then start |
| Service fails after reboot | Recovery Console; systemctl list-jobs | Reinspect custom services, unit conditions and logs |
13. Minimum command checklist for future SSH recovery
# Diagnostic: run through GoDaddy console as root.
cat /etc/os-release
/usr/sbin/sshd -t
ss -tlnp | grep ':22'
systemctl status ssh --no-pager -l
systemctl list-jobs --no-pager
journalctl -u ssh -b -n 50 --no-pager
# Only when /run/sshd is missing:
install -d -m 0755 -o root -g root /run/sshd
/usr/sbin/sshd -t
# Check for this VPS-specific deadlock, do not assume it elsewhere:
systemctl status sshd-host-keys.service --no-pager -l
systemctl cat sshd-host-keys.service
# Once the blocker and any standalone port-22 conflict have been fixed:
systemctl reset-failed ssh.service
systemctl enable ssh.service
systemctl start ssh.service
systemctl is-active ssh.service
systemctl is-enabled ssh.service
14. Final lessons learned
Connection refusedmeans the TCP service is unavailable, not that the password is incorrect.enabledis not the same asactive. The SSH service must be both.- Always inspect queued systemd jobs when
systemctl starthangs. The hidden blocker here was a custom boot-time host-key unit. - Avoid
dpkg-reconfigure openssh-serveras a routine boot-time precondition for starting SSH. It can invoke package scripts that restart SSH and create a dependency deadlock. - Do not regenerate existing SSH host keys unless there is a specific reason. Changing them unnecessarily breaks established host identities.
/usr/sbin/sshdis an emergency workaround, not the managed final state. A manual daemon may occupy port 22 and block systemd’s instance./run/sshdis a runtime directory.RuntimeDirectory=sshdin a healthy systemd service is the primary way to create it.- Use GoDaddy Recovery Console as the safety net before killing the SSH listener or rebooting.
- Verify a new remote SSH login and a known post-reboot
active/enabledstate before declaring full reboot persistence.
Sources and provenance
- Actual incident: User-provided GoDaddy VPS shell output and console transcripts from 3 November 2026; includes custom unit, process tree, SSH errors, exact final commands and results.
- Official Ubuntu upgrade guide: https://ubuntu.com/server/docs/how-to/software/upgrade-your-release/
- Official GoDaddy Recovery Console: https://www.godaddy.com/en/help/use-the-recovery-console-for-vps-hosting-32064
- Official GoDaddy Rescue Mode: https://www.godaddy.com/en/help/boot-my-vps-hosting-in-rescue-mode-32175
- systemd runtime directory documentation: https://www.freedesktop.org/software/systemd/man/256/systemd.exec.html