{"id":3520,"date":"2026-11-03T19:02:08","date_gmt":"2026-11-03T19:02:08","guid":{"rendered":"https:\/\/sreschool.com\/blog\/?p=3520"},"modified":"2026-11-03T19:02:10","modified_gmt":"2026-11-03T19:02:10","slug":"godaddy-vps-ubuntu-20-04-%e2%86%92-22-04-upgrade-ssh-recovery-root-cause-analysis-permanent-fix","status":"publish","type":"post","link":"https:\/\/sreschool.com\/blog\/godaddy-vps-ubuntu-20-04-%e2%86%92-22-04-upgrade-ssh-recovery-root-cause-analysis-permanent-fix\/","title":{"rendered":"GoDaddy VPS Ubuntu 20.04 \u2192 22.04: Upgrade, SSH Recovery, Root-Cause Analysis & Permanent Fix"},"content":{"rendered":"\n

Case study:<\/strong> VPS 184.168.122.73<\/code> \u2022 Host ip-184-168-122-73<\/code> \u2022 Login user dsbokteam<\/code> \u2022 Recovery work performed as root<\/code> \u2022 Incident observed 3 November 2026 (server UTC).
Audience:<\/strong> Linux administrators managing GoDaddy VPS servers and other Ubuntu servers.
Result:<\/strong> ssh.service<\/code> successfully became active (running)<\/code> and enabled<\/code> under systemd. A controlled post-reboot check later showed active<\/code> \/ enabled<\/code>, but the transcript does not explicitly establish whether a reboot occurred immediately before that check.<\/p>\n\n\n\n

\n

Scope and evidence:<\/strong> The SSH incident, exact errors, investigated services, and final working commands are reconstructed from the supplied console logs. The original 20.04 \u2192 22.04 upgrade command transcript was not supplied; the pre-upgrade and release-upgrade procedure<\/em> below is a recommended repeatable workflow, not a claim about commands executed in this incident<\/strong>. Do not run an upgrade again on a server already on 22.04.<\/p>\n<\/blockquote>\n\n\n\n

1. Executive summary<\/h2>\n\n\n\n

The Ubuntu upgrade was followed by a loss of remote SSH access:<\/p>\n\n\n\n

SSH Connecting to 184.168.122.73\nConnection refused (os error 61)\n<\/code><\/pre>\n\n\n\n
Initially no process listened on TCP port 22. A manual attempt to check the SSH configuration reported:<\/p>\n\n\n\n
Missing privilege separation directory: \/run\/sshd\n<\/code><\/pre>\n\n\n\n
Creating \/run\/sshd<\/code> and launching \/usr\/sbin\/sshd<\/code> restored temporary access but not automatic startup. ssh.service<\/code> was enabled but inactive<\/code>; systemctl start ssh.service<\/code> hung and timed out with code 124<\/code>.<\/p>\n\n\n\n
The underlying culprit was a custom<\/strong> \/etc\/systemd\/system\/sshd-host-keys.service<\/code> with Before=ssh.service<\/code> that executed dpkg-reconfigure --frontend=noninteractive openssh-server<\/code>. The package’s post-installation script tried to systemctl restart ssh.service<\/code>, which was still waiting for the custom service to finish: a circular wait. After the custom service was disabled and its stuck job cleared, another problem appeared: the manually launched sshd<\/code> occupied port 22, causing systemd’s sshd<\/code> to fail with Address already in use<\/code>. The manual listener was stopped safely, and systemd started the managed daemon successfully.<\/p>\n\n\n\n
2. Environment and recovery prerequisites<\/h2>\n\n\n\n
Field<\/th>Incident value<\/th><\/tr><\/thead>
Provider<\/td>GoDaddy VPS Hosting<\/td><\/tr>
Public IP<\/td>184.168.122.73<\/code><\/td><\/tr>
OS change<\/td>Ubuntu 20.04 LTS \u2192 Ubuntu 22.04 LTS<\/td><\/tr>
Normal SSH user<\/td>dsbokteam<\/code><\/td><\/tr>
Privileged shell<\/td>sudo -s<\/code> to root<\/code><\/td><\/tr>
Systemd SSH unit<\/td>ssh.service<\/code><\/td><\/tr>
Problematic extra unit<\/td>\/etc\/systemd\/system\/sshd-host-keys.service<\/code><\/td><\/tr>
SSH port<\/td>TCP 22<\/code><\/td><\/tr>
Working managed SSH PID in final incident log<\/td>7275<\/code> (will change after restart)<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
Safety prerequisites before changing remote access:<\/strong> Maintain an independent GoDaddy Recovery Console session; verify the VPS has a current restore point\/backup; do not remove SSH host keys; do not flush firewalls; do not reboot while SSH cannot start under systemd. A shell over SSH is not a substitute for out-of-band recovery access.<\/p>\n\n\n\n
GoDaddy Recovery Console versus Rescue Mode<\/h3>\n\n\n\n
    \n
  1. Sign in to GoDaddy \u2192\u00a0My Products \/ Servers<\/strong>\u00a0\u2192\u00a0Manage<\/strong>\u00a0the VPS.<\/li>\n\n\n\n
  2. Choose\u00a0Server Actions \u2192 Recovery Console<\/strong>.<\/li>\n\n\n\n
  3. Log in with a valid\u00a0Linux server account and password<\/strong>\u00a0(not the GoDaddy web-account password). The working user in this incident was\u00a0dsbokteam<\/code>\u00a0and the root shell was obtained with\u00a0sudo -s<\/code>.<\/li>\n\n\n\n
  4. Rescue Mode<\/strong>\u00a0is different: it boots into a rescue environment and GoDaddy provides temporary credentials. Use only if normal boot\/console recovery is insufficient; it interrupts hosted services.<\/li>\n<\/ol>\n\n\n\n
    Provider references: https:\/\/www.godaddy.com\/en\/help\/use-the-recovery-console-for-vps-hosting-32064<\/a> and https:\/\/www.godaddy.com\/en\/help\/boot-my-vps-hosting-in-rescue-mode-32175<\/a><\/p>\n\n\n\n
    3. Recommended pre-upgrade checklist (reference procedure; not verified incident history)<\/h2>\n\n\n\n
    Do this before<\/strong> a future Ubuntu release upgrade, in a scheduled maintenance window.<\/p>\n\n\n\n
    # Record current OS, kernel, system state and disk availability.<\/em>\ncat \/etc\/os-release\nuname -a\ndf -h\nfree -h\nsystemctl --failed --no-pager\nss -tlnp\n\n# Inspect custom services and SSH startup configuration.<\/em>\nsystemctl cat ssh.service\nsystemctl list-unit-files '*ssh*'\nls -la \/etc\/systemd\/system\/*ssh* 2>\/dev\/null\nsudo \/usr\/sbin\/sshd -t\n\n# Record installed packages and important configuration.<\/em>\nsudo install -d -m 0700 \/root\/pre-upgrade-records\nsudo dpkg-query -W > \/root\/pre-upgrade-records\/packages.txt\nsudo cp -a \/etc\/ssh \/root\/pre-upgrade-records\/ssh-config-backup\nsudo cp -a \/etc\/apt \/root\/pre-upgrade-records\/apt-config-backup\n<\/code><\/pre>\n\n\n\n
    Security:<\/strong> \/etc\/ssh<\/code> may contain private host keys. Keep that backup restricted to root, preferably encrypted\/off-server, and do not publish it. In addition, take a provider snapshot or separately verified full backup of application data, databases, configurations and volumes. Verify recovery access before<\/em> changing the OS.<\/p>\n\n\n\n
    Update the current Ubuntu release fully and install the official upgrader (run only while still on the old<\/strong> release):<\/p>\n\n\n\n
    sudo apt update\nsudo apt dist-upgrade -o APT::Get::Always-Include-Phased-Updates=true\nsudo apt install update-manager-core\nsudo grep '^Prompt=' \/etc\/update-manager\/release-upgrades\n<\/code><\/pre>\n\n\n\n
    For LTS-to-LTS upgrades, Prompt=lts<\/code> is appropriate. Read package prompts carefully, especially if \/etc\/ssh\/sshd_config<\/code> or third-party repositories have local customizations. Do not<\/strong> blindly overwrite an existing SSH configuration.<\/p>\n\n\n\n
    4. Recommended Ubuntu 20.04 \u2192 22.04 upgrade procedure (reference only)<\/h2>\n\n\n\n
    The official server upgrade mechanism is:<\/p>\n\n\n\n
    sudo do-release-upgrade\n<\/code><\/pre>\n\n\n\n
    Run only on the pre-upgrade server with backups and recovery access ready. Read the proposed package removals\/replacements before proceeding. Ubuntu’s release upgrader handles configuration questions interactively and typically runs in a recoverable GNU screen session for remote upgrades. Do not use -d<\/code> for a routine production LTS upgrade.<\/p>\n\n\n\n
    At the end of a successful upgrade, a reboot is normally required. The release upgrade is not complete until that reboot. See Ubuntu’s official instructions: https:\/\/ubuntu.com\/server\/docs\/how-to\/software\/upgrade-your-release\/<\/a><\/p>\n\n\n\n
    Before rebooting, check for unexpected custom units such as sshd-host-keys.service<\/code>.<\/strong> On this VPS it was a custom service outside Ubuntu’s regular ssh.service<\/code> implementation and blocked boot-time startup.<\/p>\n\n\n\n
    5. Incident chronology: actual observed errors and meaning<\/h2>\n\n\n\n
    Stage<\/th>Exact observation<\/th>Technical meaning<\/th><\/tr><\/thead>
    SSH unavailable<\/td>Connection refused (os error 61)<\/code><\/td>TCP connection refused before authentication; not a bad password<\/td><\/tr>
    Listener check<\/td>`ss -tlnp<\/td>grep ‘:22’` returned nothing<\/td><\/tr>
    SSH validation<\/td>Missing privilege separation directory: \/run\/sshd<\/code><\/td>Required runtime directory was absent<\/td><\/tr>
    Manual fallback<\/td>\/usr\/sbin\/sshd<\/code> worked<\/td>SSH binary could run outside systemd<\/td><\/tr>
    Enable status<\/td>enabled<\/code> \/ inactive<\/code><\/td>Configured for boot but not actually running as a managed service<\/td><\/tr>
    Managed startup<\/td>timeout 20s systemctl start ssh.service<\/code> \u2192 Exit code: 124<\/code><\/td>Startup waited beyond 20 seconds<\/td><\/tr>
    Systemd queue<\/td>sshd-host-keys.service start running<\/code> and ssh.service start waiting<\/code><\/td>An earlier ordering\/dependency blocker was active<\/td><\/tr>
    Custom unit<\/td>Before=ssh.service<\/code>; ExecStart=dpkg-reconfigure --frontend=noninteractive openssh-server<\/code><\/td>Custom host-key job reconfigured the package at boot<\/td><\/tr>
    Reconfiguration child<\/td>deb-systemd-invoke restart rescue-ssh.target ssh.service<\/code><\/td>Host-key job tried to restart the service it preceded<\/td><\/tr>
    After unblocking<\/td>Bind to port 22 ... Address already in use<\/code>; fatal: Cannot bind any address.<\/code><\/td>Manual sshd<\/code> had become the remaining port conflict<\/td><\/tr>
    Final result<\/td>ssh.service: active (running)<\/code>, enabled<\/code>, systemd-managed PID 7275<\/td>Managed SSH startup succeeded<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
    Root cause: ordering and restart deadlock<\/h3>\n\n\n\n
    systemd starts sshd-host-keys.service\n        |\n        v\nExecStart: dpkg-reconfigure openssh-server\n        |\n        v\nopenssh-server.postinst calls systemctl restart ssh.service\n        |\n        v\nssh.service is WAITING for sshd-host-keys.service\n        |\n        +------ circular wait \/ stalled boot transaction ------+\n<\/code><\/pre>\n\n\n\n
    The file \/etc\/systemd\/system\/sshd-host-keys.service<\/code> contained:<\/p>\n\n\n\n
    [Unit]\nDescription=Generates ssh host keys at boot time\nDocumentation=man:sshd(5)\nBefore=ssh.service\n\n[Service]\nType=oneshot\nRemainAfterExit=yes\nUser=root\nExecStart=dpkg-reconfigure --frontend=noninteractive openssh-server\n\n[Install]\nWantedBy=multi-user.target\n<\/code><\/pre>\n\n\n\n
    This unit should not be assumed to exist on every Ubuntu server; it was specific to this VPS<\/strong>. In the incident, SSH host keys already existed (\/etc\/ssh\/ssh_host_rsa_key<\/code>, ssh_host_ecdsa_key<\/code>, ssh_host_ed25519_key<\/code>). No regeneration was necessary.<\/p>\n\n\n\n
    6. First temporary recovery: repair \/run\/sshd<\/code><\/h2>\n\n\n\n
    From GoDaddy Recovery Console, authenticate as the Linux user and get root privileges:<\/p>\n\n\n\n
    sudo -s\n<\/code><\/pre>\n\n\n\n
    Investigate the server:<\/p>\n\n\n\n
    cat \/etc\/os-release\nps -p 1 -o pid,comm,args\nps aux | grep '[s]shd'\nss -tlnp | grep ':22'\ndpkg -l openssh-server\n\/usr\/sbin\/sshd -t\n<\/code><\/pre>\n\n\n\n
    When sshd -t<\/code> reports a missing runtime directory:<\/p>\n\n\n\n
    install -d -m 0755 -o root -g root \/run\/sshd\n\/usr\/sbin\/sshd -t\necho $?\n# Expected: 0<\/em>\n<\/code><\/pre>\n\n\n\n
    The temporary<\/em> emergency workaround that succeeded was:<\/p>\n\n\n\n
    \/usr\/sbin\/sshd\nss -tlnp | grep ':22'\n<\/code><\/pre>\n\n\n\n
    This is not a permanent boot fix.<\/strong> \/usr\/sbin\/sshd<\/code> starts a daemon without making systemd own the process. Once systemd tries to start its own instance, the manually created listener can conflict on port 22.<\/p>\n\n\n\n
    7. Investigate systemd rather than repeating a hanging command<\/h2>\n\n\n\n
    # Show the service without letting output enter a pager.<\/em>\nsystemctl status ssh.service --no-pager -l\nsystemctl is-enabled ssh.service\nsystemctl is-active ssh.service\n\n# Bound startup attempts so the console does not hang indefinitely.<\/em>\ntimeout 20s systemctl start ssh.service\necho \"Exit code: $?\"\n\n# Identify the queued job or dependency.<\/em>\nsystemctl list-jobs --no-pager\njournalctl -u ssh.service -b -n 50 --no-pager\nsystemctl cat ssh.service\n<\/code><\/pre>\n\n\n\n
    A timeout<\/code> exit code of 124<\/code> means the command timed out; it does not<\/strong> by itself prove sshd<\/code> has a corrupt configuration. In this case, systemctl list-jobs<\/code> exposed the real blocker:<\/p>\n\n\n\n
    183 sshd-host-keys.service             start running\n120 ssh.service                       start waiting\n175 systemd-update-utmp-runlevel.service start waiting\n2   multi-user.target                 start waiting\n1   graphical.target                  start waiting\n<\/code><\/pre>\n\n\n\n
    Inspect that specific unit:<\/p>\n\n\n\n
    systemctl status sshd-host-keys.service --no-pager -l\nsystemctl cat sshd-host-keys.service\njournalctl -u sshd-host-keys.service -b -n 50 --no-pager\nps -ef | grep -E 'dpkg-reconfigure|openssh-server.postinst|systemctl.*ssh'\nls -l \/etc\/ssh\/ssh_host_*_key\n<\/code><\/pre>\n\n\n\n
    The process tree showed dpkg-reconfigure<\/code>, openssh-server.postinst<\/code>, deb-systemd-invoke restart ... ssh.service<\/code> and systemctl ... restart ssh.service<\/code>. That confirmed the startup cycle.<\/p>\n\n\n\n
    8. Permanent repair, part A: remove the harmful boot-time dependency<\/h2>\n\n\n\n
    Apply this only to the confirmed custom unit above, not to Ubuntu’s normal ssh.service<\/code>.<\/strong> Keep GoDaddy Recovery Console open.<\/p>\n\n\n\n
    # Prevent the custom unit from being started during future boots.<\/em>\nsystemctl disable sshd-host-keys.service\n\n# Inspect the active job, if any, before stopping it.<\/em>\nsystemctl status sshd-host-keys.service --no-pager -l\nsystemctl list-jobs --no-pager\n<\/code><\/pre>\n\n\n\n
    In this incident the custom unit was stuck inside its own package reconfiguration. The recovery procedure was to stop that specific<\/em> custom unit and, if it remained stuck, terminate only the processes in its cgroup:<\/p>\n\n\n\n
    # If stuck, terminate only the custom service's child processes.<\/em>\nsystemctl kill --kill-who=all --signal=SIGTERM sshd-host-keys.service\nsystemctl stop sshd-host-keys.service\nsystemctl daemon-reload\n<\/code><\/pre>\n\n\n\n
    Do not forcibly kill unrelated package upgrades. Confirm package manager jobs are not actively performing a different upgrade before intervening. If systemctl stop<\/code> hangs, inspect jobs again; preserve the console connection.<\/p>\n\n\n\n
    Check the result:<\/p>\n\n\n\n
    systemctl is-enabled sshd-host-keys.service\n# Observed final result: disabled<\/em>\nsystemctl list-jobs --no-pager\n# Observed final result: No jobs running.<\/em>\n<\/code><\/pre>\n\n\n\n
    Optional:<\/strong> If the unit repeatedly gets reactivated by another component, investigate that dependency. systemctl mask sshd-host-keys.service<\/code> can prevent starts, but was not required<\/strong> for the successful final state recorded here. Be cautious with provider-customized units and document any local changes.<\/p>\n\n\n\n
    9. Permanent repair, part B: move SSH from a manual process to systemd<\/h2>\n\n\n\n
    Once the dependency cycle was cleared, the managed start failed with:<\/p>\n\n\n\n
    error: Bind to port 22 on 0.0.0.0 failed: Address already in use.\nerror: Bind to port 22 on :: failed: Address already in use.\nfatal: Cannot bind any address.\n<\/code><\/pre>\n\n\n\n
    This was because the manually started SSH listener owned TCP port 22. A later check also showed \/run\/sshd<\/code> missing again. That can happen because ssh.service<\/code> declares RuntimeDirectory=sshd<\/code> and systemd ordinarily removes the runtime directory when the service stops; it is not proof that the filesystem is corrupted.<\/p>\n\n\n\n
    Exact working recovery sequence (run from independent GoDaddy console):<\/strong><\/p>\n\n\n\n
    # 1. Recreate the runtime directory and validate configuration.<\/em>\ninstall -d -m 0755 -o root -g root \/run\/sshd\n\/usr\/sbin\/sshd -t\necho $?\n# Observed: 0<\/em>\n\n# 2. Verify the custom job is disabled and no queued jobs remain.<\/em>\nsystemctl is-enabled sshd-host-keys.service\nsystemctl list-jobs --no-pager\n# Observed: disabled; No jobs running.<\/em>\n\n# 3. Identify the *current* listener. Do not assume an old PID is still valid.<\/em>\nss -tlnp | grep ':22'\nps -p 6534 -o pid,ppid,args\n# In the incident, PID 6534 was the standalone sshd listener.<\/em>\n\n# 4. Terminate ONLY the verified standalone listener, from Recovery Console.<\/em>\nkill -TERM 6534\n\n# 5. Clear systemd's earlier failed state and start managed SSH.<\/em>\nsystemctl reset-failed ssh.service\ntimeout 20s systemctl start ssh.service\necho \"Exit code: $?\"\n# Observed: Exit code: 0<\/em>\n\n# 6. Verify startup and boot enablement.<\/em>\nsystemctl status ssh.service --no-pager -l\nsystemctl is-enabled ssh.service\nsystemctl is-active ssh.service\n# Observed: enabled; active<\/em>\n<\/code><\/pre>\n\n\n\n
    Important:<\/strong> 6534<\/code> was a real incident PID, not a reusable constant. For a different server or later boot, discover and verify the current listener before sending a signal. If your only access is SSH, do not<\/strong> kill its listener without an independent Recovery Console session. The final managed SSH process was PID 7275<\/code> and logged:<\/p>\n\n\n\n
    Started OpenBSD Secure Shell server.\nServer listening on 0.0.0.0 port 22.\nServer listening on :: port 22.\n<\/code><\/pre>\n\n\n\n
    10. Why \/run\/sshd<\/code> vanished and how it is managed at boot<\/h2>\n\n\n\n
    Ubuntu’s ssh.service<\/code> unit on this VPS includes:<\/p>\n\n\n\n
    ExecStartPre=\/usr\/sbin\/sshd -t\nExecStart=\/usr\/sbin\/sshd -D $SSHD_OPTS\nType=notify\nRuntimeDirectory=sshd\nRuntimeDirectoryMode=0755\nRestart=on-failure\n<\/code><\/pre>\n\n\n\n
    Normal behavior:<\/strong> When systemd starts ssh.service<\/code>, it creates \/run\/sshd<\/code>. When that service stops, systemd may remove the directory. The directory is temporary and should not be made persistent by storing it under a different filesystem.<\/p>\n\n\n\n
    During troubleshooting, this additional tmpfiles rule was created:<\/p>\n\n\n\n
    cat > \/etc\/tmpfiles.d\/sshd.conf <<'RULE'\nd \/run\/sshd 0755 root root -\nRULE\nsystemd-tmpfiles --create \/etc\/tmpfiles.d\/sshd.conf\n<\/code><\/pre>\n\n\n\n
    This is a permissible additional boot-time safeguard, but not the main fix<\/strong>: correct ssh.service<\/code> startup already creates \/run\/sshd<\/code> automatically. The main fix was clearing the custom service deadlock and moving SSH under systemd management.<\/p>\n\n\n\n
    Reference: https:\/\/www.freedesktop.org\/software\/systemd\/man\/256\/systemd.exec.html<\/a><\/p>\n\n\n\n
    11. Final validation and controlled reboot<\/h2>\n\n\n\n
    Before rebooting, verify all of these:<\/p>\n\n\n\n
    # Should show active \/ enabled.<\/em>\nsystemctl is-active ssh.service\nsystemctl is-enabled ssh.service\n\n# Should show disabled (or masked if deliberately masked).<\/em>\nsystemctl is-enabled sshd-host-keys.service\n\n# Should show no blocked jobs.<\/em>\nsystemctl list-jobs --no-pager\n\n# Should show the managed daemon listening on 22.<\/em>\nsystemctl show ssh.service -p ActiveState -p SubState -p MainPID -p RuntimeDirectory\nss -tlnp | grep ':22'\n\n# Should validate cleanly.<\/em>\n\/usr\/sbin\/sshd -t\n<\/code><\/pre>\n\n\n\n
    Additional overall checks:<\/p>\n\n\n\n
    systemctl --failed --no-pager\ndpkg --audit\ndf -h\n<\/code><\/pre>\n\n\n\n
    From a new terminal on your Mac<\/strong> (use your established SSH authentication method):<\/p>\n\n\n\n
    ssh dsbokteam@184.168.122.73\n<\/code><\/pre>\n\n\n\n
    Only after a fresh connection succeeds<\/strong>, your backup\/recovery console is available, and ssh.service<\/code> is active\/enabled, conduct a controlled reboot during a maintenance window:<\/p>\n\n\n\n
    sudo reboot\n<\/code><\/pre>\n\n\n\n
    After the server returns:<\/p>\n\n\n\n
    systemctl is-active ssh\nsystemctl is-enabled ssh\nsystemctl is-enabled sshd-host-keys.service\nsystemctl list-jobs --no-pager\nss -tlnp | grep ':22'\n<\/code><\/pre>\n\n\n\n
    Observed at end of user log:<\/strong> systemctl is-active ssh<\/code> \u2192 active<\/code>; systemctl is-enabled ssh<\/code> \u2192 enabled<\/code>. The log does not explicitly say whether this check followed a reboot, so do not treat reboot persistence as independently proved until you verify it after a known reboot.<\/p>\n\n\n\n
    12. Additional upgrade health checks and only-if-needed repairs<\/h2>\n\n\n\n
    After a major OS upgrade, assess applications independently from SSH:<\/p>\n\n\n\n
    cat \/etc\/os-release\nuname -r\nsudo systemctl --failed --no-pager\nsudo dpkg --audit\nsudo journalctl -b -p err --no-pager -n 100\nsudo ss -tlnp\nsudo df -h\n<\/code><\/pre>\n\n\n\n
    Check only services actually installed on the server, for example nginx<\/code>, apache2<\/code>, mysql<\/code>, mariadb<\/code>, php*-fpm<\/code>, firewalls, or monitoring agents. Third-party repositories may have been disabled during the distribution upgrade; inspect them before re-enabling compatibility-sensitive packages.<\/p>\n\n\n\n
    If dpkg --audit<\/code> reports half-configured packages and no other apt\/dpkg\/release-upgrader job is running<\/strong>, the general Ubuntu repair tools are:<\/p>\n\n\n\n
    sudo dpkg --configure -a\nsudo apt --fix-broken install\n<\/code><\/pre>\n\n\n\n
    However:<\/strong> Because openssh-server<\/code> reconfiguration specifically triggered the deadlock in this case, do not run those package-repair commands blindly while a problematic sshd-host-keys.service<\/code> is still running or while SSH recovery is incomplete. Inspect any proposed changes and keep the Recovery Console available.<\/p>\n\n\n\n
    Troubleshooting matrix<\/h3>\n\n\n\n
    Symptom<\/th>Check<\/th>Correct response<\/th><\/tr><\/thead>
    SSH Connection refused<\/code><\/td>ss -tlnp | grep ':22'<\/code><\/td>Check service\/listener and port, not password first<\/td><\/tr>
    sshd -t<\/code> says missing \/run\/sshd<\/code><\/td>ls -ld \/run\/sshd<\/code><\/td>install -d -m 0755 -o root -g root \/run\/sshd<\/code>; then sshd -t<\/code><\/td><\/tr>
    systemctl start ssh<\/code> hangs<\/td>systemctl list-jobs<\/code><\/td>Find waiting unit; inspect custom dependencies<\/td><\/tr>
    ssh.service<\/code> enabled but inactive<\/td>systemctl status ssh<\/code><\/td>Enabled is only boot configuration; fix startup and retest<\/td><\/tr>
    sshd-host-keys.service<\/code> stuck<\/td>systemctl cat<\/code> and status<\/code><\/td>Identify dpkg-reconfigure<\/code> \/ recursive SSH restart; disable confirmed custom unit<\/td><\/tr>
    Address already in use<\/code> on port 22<\/td>ss -tlnp | grep ':22'<\/code><\/td>Transition from verified standalone listener to systemd with out-of-band access<\/td><\/tr>
    ssh.service<\/code> failed after a former error<\/td>journalctl -u ssh -b<\/code><\/td>Fix root problem, systemctl reset-failed ssh<\/code>, then start<\/td><\/tr>
    Service fails after reboot<\/td>Recovery Console; systemctl list-jobs<\/code><\/td>Reinspect custom services, unit conditions and logs<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
    13. Minimum command checklist for future SSH recovery<\/h2>\n\n\n\n
    # Diagnostic: run through GoDaddy console as root.<\/em>\ncat \/etc\/os-release\n\/usr\/sbin\/sshd -t\nss -tlnp | grep ':22'\nsystemctl status ssh --no-pager -l\nsystemctl list-jobs --no-pager\njournalctl -u ssh -b -n 50 --no-pager\n\n# Only when \/run\/sshd is missing:<\/em>\ninstall -d -m 0755 -o root -g root \/run\/sshd\n\/usr\/sbin\/sshd -t\n\n# Check for this VPS-specific deadlock, do not assume it elsewhere:<\/em>\nsystemctl status sshd-host-keys.service --no-pager -l\nsystemctl cat sshd-host-keys.service\n\n# Once the blocker and any standalone port-22 conflict have been fixed:<\/em>\nsystemctl reset-failed ssh.service\nsystemctl enable ssh.service\nsystemctl start ssh.service\nsystemctl is-active ssh.service\nsystemctl is-enabled ssh.service\n<\/code><\/pre>\n\n\n\n
    14. Final lessons learned<\/h2>\n\n\n\n
      \n
    1. Connection refused<\/code>\u00a0means the TCP service is unavailable, not that the password is incorrect.<\/strong><\/li>\n\n\n\n
    2. enabled<\/code>\u00a0is not the same as\u00a0active<\/code>.<\/strong>\u00a0The SSH service must be both.<\/li>\n\n\n\n
    3. Always inspect queued systemd jobs when\u00a0systemctl start<\/code>\u00a0hangs.<\/strong>\u00a0The hidden blocker here was a custom boot-time host-key unit.<\/li>\n\n\n\n
    4. Avoid\u00a0dpkg-reconfigure openssh-server<\/code>\u00a0as a routine boot-time precondition for starting SSH.<\/strong>\u00a0It can invoke package scripts that restart SSH and create a dependency deadlock.<\/li>\n\n\n\n
    5. Do not regenerate existing SSH host keys unless there is a specific reason.<\/strong>\u00a0Changing them unnecessarily breaks established host identities.<\/li>\n\n\n\n
    6. \/usr\/sbin\/sshd<\/code>\u00a0is an emergency workaround, not the managed final state.<\/strong>\u00a0A manual daemon may occupy port 22 and block systemd’s instance.<\/li>\n\n\n\n
    7. \/run\/sshd<\/code>\u00a0is a runtime directory.<\/strong>\u00a0RuntimeDirectory=sshd<\/code>\u00a0in a healthy systemd service is the primary way to create it.<\/li>\n\n\n\n
    8. Use GoDaddy Recovery Console as the safety net<\/strong>\u00a0before killing the SSH listener or rebooting.<\/li>\n\n\n\n
    9. Verify a new remote SSH login and a known post-reboot\u00a0active<\/code>\/enabled<\/code>\u00a0state<\/strong>\u00a0before declaring full reboot persistence.<\/li>\n<\/ol>\n\n\n\n
      Sources and provenance<\/h2>\n\n\n\n