Linux – ssh immediately disconnects “Write failed: broken pipe”

linuxssh

When I try to log in to a remote machine via ssh, I immediately get disconnected with "Write failed: broken pipe". I've tried logging in via mosh instead and the mosh client freezes.

I've tried googling around and the only answers I can come up with are related to timeouts (for example, this Unix stack exchange question). I'm pretty certain it's not a timeout related issue since I made no personal configuration changes since last night (when it was last working) and other users are able to log in just fine.

Running ssh with -vvv (very very verbose logs) revealed nothing helpful.

I already solved my specific issue (see my answer below), but I was wondering what other problems might cause the issue (ssh disconnecting immediately stating only "Write failed: broken pipe")?

I further wonder if sshd/mosh/etc could report on being unable to spawn a session to help diagnose problems?

Best Answer

So, in my specific case, I had left a screen open running gdb attached to a process trying to reproduce an issue a customer was having related to our product spawning too many threads. Well apparently the issue had been reproduced overnight and my user had reached the system's user thread limit.

So sshd could not spawn a new process under my user due to the process and thread limit. Ultimately what we had to do was to log in with a different user and kill my running screen, gdb, and the process being debugged. This freed up process and thread resources allowing me to finally reconnect.

Related Question