Troubleshooting mod_cluster

So you are trying to come to grips with mod_cluster and JBoss/WildFly but your clustering still doesn’t work? in this article I’ll discuss about troubleshooting some common issues. 

Mod_cluster Advice #1

The first thing you need obviously to check is that you are running a version of mod_cluster which is compatible with your machine and with the version of Apache you have installed.
If you enter the downloads area you will see that mod_cluster binaries are available in two formats:

linux-x86_64 mod_cluster binaries  
linux-x86_64 httpd+ssl binaries

The first one includes just the mod_cluster libraries to be installed on Apache. The second one includes as well an Apache httpd server. Hence, if installing mod_cluster modules fails I’d suggest to try with the second option (embedded apache) or download the embedded version and check which Apache release is used then:

jbosseap sbin # ./httpd -v
Server version: Apache/2.2.21 (Unix)
Server built:   Feb  9 2012 22:21:06

Mod_cluster Advice #2

Even if Apache starts successfully, it can be that some of Apache modules are not running correctly. So, provided that you have installed the mod_cluster_manager, include AllowDisplay On in your configuration, as in the following example:

Listen 127.0.0.1:6666

<VirtualHost 127.0.0.1:6666>  
      <Location />
          Order deny,allow
          Deny from all
          Allow from 127.0.0
      </Location>

    <Location /mod_cluster_manager>
       SetHandler mod_cluster-manager
       Order deny,allow
       Deny from all
       Allow from 127.0.0
    </Location>      

      KeepAliveTimeout 60
      MaxKeepAliveRequests 0
      EnableMCPMReceive On
      AllowDisplay On
      ManagerBalancerName mycluster
      ServerAdvertise On
      
</VirtualHost>

The following picture shows a correctly started Apache server with mod_cluster:

Mod_cluster Advice #3
Check your firewall!
If you are running a Linux/Unix box it’s likely that you have stricter security requirements. So chances are that iptables or selinux are blocking your messages. You have several option, start by disabling iptables and selinux to see if that’s the problem:

# /etc/init.d/iptables stop

Making Selinux permissive requires changing as root a configuration file. In Fedora Core and RedHat Enterprise, edit /etc/selinux/config and you will see some lines like this:

SELINUX=enforcing
# SELINUXTYPE= can take one of these two values:
# targeted - Only targeted network daemons are protected.
# strict - Full SELinux protection.

… just change SELINUX=enforcing to SELINUX=permissive, and you’re done. Reboot

If mod_cluster works correctly consider configuring your Linux security policies to allow certain traffic like the following ones:

sudo iptables -I INPUT 1 -p udp -d 224.0.1.0/24 -j ACCEPT -m comment --comment "allow mod_cluster multicast udp traffic"
sudo iptables -I INPUT 2 -p udp -d 224.0.0.0/4 -j ACCEPT -m comment --comment "allow clustered JBoss multicast udp traffic"
sudo iptables -I INPUT 3 -p udp -s 10.0.1.0/24 -j ACCEPT -m comment --comment "allow node to node udp traffic"
sudo iptables -I INPUT 4 -p tcp -s 10.0.1.0/24 -j ACCEPT -m comment --comment "allow node to node tcp traffic"
sudo /etc/init.d/iptables save

Mod_cluster Advice #4
Failures in multicast is a common source of issues for networking tools like mod_cluster. Luckily, there are plenty of tools which can repidly check if you are correctly sending and receiving multicast messages across your cluster. Personally I find quite useful the little Advertize.java class which can be downloaded from:
https://github.com/modcluster/mod_cluster/blob/master/test/java/Advertize.java

Once compiled and executed as follows:

java Advertize 224.0.1.105 23364

or:

java Advertize 224.0.1.105 23364 10.33.144.3

You should see a log packets which are going through the multicast address (and maybe IP Address) that you have included as argument. A successful multicast communication should output the following log:

received from /192.168.118.132:23364
received: HTTP/1.0 200 OK
Date: Thu, 04 Sep 2014 08:35:56 GMT
Sequence: 20
Digest: 9419b474ad425566b4cc4b094f3b8e9f
Server: fe296f09-b1f2-4b17-87f2-95c031812814
X-Manager-Address: 127.0.0.1:6666
X-Manager-Url: /fe296f09-b1f2-4b17-87f2-95c031812814
X-Manager-Protocol: http
X-Manager-Host: localhost

If multicast does not work do not despair! you can still rely on unicast messages to communicate between Apache and JBoss/WildFly. Include in your mod_cluster configuration the proxy-list of Apache servers as in the following example:
  <mod-cluster-config advertise-socket=”modcluster” connector=”ajp” proxy-list=”127.0.0.1:6666″>

Mod_cluster Advice #5
Check your configuration! Have you included a custom AdvertiseGroup in your httpd.conf ?
Apache AdvertiseGroup 224.0.1.106:23364
Then you should configure mod_cluster subsystem to match the same IP:Port

    <socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
                . . . . . .
                <socket-binding name="modcluster" port="0" multicast-address="224.0.1.106" multicast-port="23364"/>

Have you included the following directive in httpd.conf ?

EnableMCPMReceive On

This directive allow the Apache VirtualHost to receive the MCPM from the nodes. You need one EnableMCPMReceive in your httpd configuration to allow mod_cluster to work.

If everything fails, your last (not least!) option will be of course checking Apache server logs. A successful boot of mod_cluster should be trace as follows:

[Thu Sep 04 10:30:20 2014] [notice] Advertise initialized for process 3786
[Thu Sep 04 10:30:20 2014] [notice] Apache/2.2.21 (Unix) mod_cluster/1.2.0.Final mod_ssl/2.2.21 OpenSSL/0.9.8r DAV/2 mod_jk/1.2.30 configured -- resuming normal operations

That’s not so much information so if you are still experiencing problems, set the Apache log level to an higher value:

LogLevel debug

This will trace all the undergoing communication between Apache and mod-cluster, as in the following excerpt:

[Thu Sep 04 10:50:48 2014] [debug] proxy_util.c(2011): proxy: ajp: has acquired connection for (127.0.0.1)
[Thu Sep 04 10:50:48 2014] [debug] proxy_util.c(2067): proxy: connecting ajp://127.0.0.1:8009/ to 127.0.0.1:8009
[Thu Sep 04 10:50:48 2014] [debug] proxy_util.c(2193): proxy: connected ajp://127.0.0.1:8009/ to 127.0.0.1:8009
[Thu Sep 04 10:50:48 2014] [debug] mod_proxy_cluster.c(773): ajp_cping_cpong: Done