My BLOG!

Server stability achieved.. for now

Ultimately it was Justin, the main guy at my provider, Awknet, that figured out what was causing apache failures.

This being a fresh install of Freebsd with cpanel/whm, meant some of the default settings, in particular the "maxclient setting" may not have been optimal for an upload site (as might have been with say, Linux, but that's just speculation).

Here's what was happenning (messages recieved from Justin):

Update:

Oh, hey, I've got an idea... it might not be that apache is crashing, but rather that you have more than 150 clients downloading from the server at once.

Imagine you have 50 people visiting your site, and 50 downloading big files already. 25 of the new 50 start to download big files too. This continues for a while, soon you have 148 people downloading big files, perhaps on slow home connections like 128kbps ISDN or something horrific. Two more people start downloading, now there are 150 clients connected. Apache puts all new requests "on hold", waiting for the old requests to finish soon. Result? People don't get to see the websites or download files.

Ways to get around this? Setup a seperate server to manage downloads. By seperate server I mean seperate webserver process, not new machine. You could setup the new webserver process to not load any extra frills like php/perl/etc. Just purely for serving files. You could also use something like thttpd.

The draw back? I don't think you can do it from cpanel, so you'd have to be familiar with the UNIX environment. And you'd have to think about how you're verifying users have permission to download files, etc.

If that's what's happening we're going to see inaccessibility happen again, but the processes will all still be running. If this happens a "ps auxw | grep apache | wc -l" command will probably show something crazy like, 150 or so. If it does - that's the problem. If it's not that many, it's something else.

 

Update:

Yep - low and behold, happening again, and;

# ps uaxw | grep apache | wc -l
    152

More than 150 apache processes running. Now I'm going to bump that limit higher (this may not be too great of an idea as it wastes memory when running full blown apache like this) - setting it to 220.

 

Update:

Already at 146;

host# ps uaxw | grep apache | wc -l
    146

Update:

host# ps uaxw | grep apache | wc -l
    185

;)

Update:

Well, until you think of a longer term solution (seperate web process for downloads, perhaps running thttpd or something other than apache? Or check into apache2 with threading support installed via cpanel, etc) I'm going to up the apache limit to 350. You're only showing 4mb of free ram at max load right now, however there is 1.3GB listed as "inactive", so, it should be able to start swapping that over instead. If things start dragging to a crawl you're going into swap and we've gotta lower that limit again, so let me know. :)

 

Update:

...229 people downloading files right now, and there's still 1.246GB of ram free, so... I'll knock the limit upto 750. Things should be fine. If anything starts acting weird just page me ([edited]) and I'll hop on and lower the values. :)

So the maxclient setting has been bumped up to 750.  Things look stable with current configurations (esp. memory and server load):

Server Load 0.19 (1 cpu) 
Memory Used 46.63 % 
Swap 0.00 % 
Disk ad4s1a (/)  3 % 

And apache isn't failing anymore! Sites are loading quick.  *whew* that's a relief!

I am going to look into rebuilding apache with thttpd (if that's even possible), or getting apache2 installed with threading support... just in case.