Infinite loop in case of @pony_os_accept() error

[slfritchie]

Today's master branch's commit 1ea9650 for https://github.com/WallarooLabs/wallaroo/blob/master/lib/wallaroo/core/data_channel/data_channel_listener.pony#L174-L187 shows that if @pony_os_accept() call fails, then there's an immediate retry. For some error conditions, this is fine, but for other conditions, such as EMFILE, this is bad.

In the case of EMFILE, the process has it its file descriptor limit. Odds are extremely high that EMFILE will happen again in this kind of tight loop. Also, the condition can last long enough that the actor's behavior call doesn't return promptly, which can occupy an entire Pony runtime scheduler thread ... and, worst case, occupy all available scheduler threads. Credit to @JONBRWN for discovering this scenario with a 30 worker Wallaroo cluster and a Linux ulimit -n value of 1024.