devbox@COMPUTEC The Computec development blog

CF Instance Hangs, “too many open files” in Log

This is really quite trivial, but may be of some interest to people who don't know too much about Linux OS tweaking. Today our dev-server instance was hanging - again. Only this time I took the liberty to make our developers wait a little longer before I restarted it, so I could actually diagnose the issue and resolve it for good. As only one of our two instances was hanging, I took a peek into its logfiles first - but couldn't find anything out of the ordinary. And the Enterprise Manager reported the instance to be running, too.

So I got the impression that the problem must be some level higher up, i.e. not inside the instance. Looking at the tail of my /opt/jrun4/logs/cfusion-event.log I got myself a winner:

21/07 11:43:02 error Too many open files
java.net.SocketException: Too many open files
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
        at java.net.ServerSocket.implAccept(ServerSocket.java:453)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at jrun.servlet.network.NetworkService.accept(NetworkService.java:356)
        at jrun.servlet.jrpp.JRunProxyService.accept(JRunProxyService.java:103)
        at jrun.servlet.jrpp.JRunProxyService.createRunnable(JRunProxyService.java:119)
        at jrunx.scheduler.ThreadPool$ThreadThrottle.createRunnable(ThreadPool.java:349)
        at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

If you know where to look, this may be easily resolved. The maximum number of open files for a certain user can be displayed and set with the ulimit -n command. There is a hard and a soft limit. The soft limit is the one currently set for the user with ulimit -n xxx, the hard limit is the max. number the user can actually set his soft limit to with the ulimit -n command. The hard limit can be displayed with ulimit -Hn.

First thing we do is edit our ColdFusion init-script (mine is /etc/init.d/coldfusion8multi) and add the following line somewhere before the actual start of jrun:

ulimit -n `ulimit -Hn`

Don't restart ColdFusion just yet as most likely both your soft and hard limits are set to 1024 - and this not being high enough is the actual reason for our problem.

You'll need to edit /etc/security/limits.conf next in order to alter the soft and hard limits. If you just want to increase the filehandle-limits for all accounts on this box, the following two lines would be sufficient:

*                soft    nofile          1024
*                hard    nofile          8192

If you just want to set it for the jrun account, replace the wildcard with the account-name. Settings for groups can be done using the @group syntax. The file is fairly well documented inside on most standard installs, so no worries. The changes in this file affect all new logins (or newly started processes), so your running processes are not affected, but after a restart, ColdFusion will have picked up the changes.

There's one issue remaining - the total number of open files for the system has a limit, too. To show the current limit, you can use cat /proc/sys/fs/file-max. You could also write to that pseudo-file in order to increase the limit, but your changes would be lost on system reboot. In order to make your settings permanent, you'll need to edit /etc/sysctl.conf. Add a line like

fs.file-max = 275140

Do a sysctl -p /etc/sysctl.conf for the changes to take effect.