Configuring JBoss EAP 6 – WildFly to use TCP for messaging

This tutorial shows how to use TCP for initial discovery and communication between cluster nodes of HornetQ servers running on JBoss EAP 6 / WildFly 8

First of all, let’s see the default mechanism used by the HornetQ messaging broker in JBoss/WildFly:

As you can see, the first step is server discovery and it’s done by JGroups, by default using UDP. Next, once that the servers have found each other, starts the communication between nodes. This happens via the Netty TCP Connections (JBoss EAP 6) or tunnelled via HTTP /WildFly).

That being said, there are two strategies for switching from UDP to TCP:

1) Rely on JGroups configuration

The simplest choice is to rely on JGroups. So, say for example that you have defined a list of static nodes via TCPPING

<subsystem xmlns="urn:jboss:domain:jgroups:2.0" default-stack="tcpping">
    <stack name="tcpping">
        <transport type="TCP" socket-binding="jgroups-tcp"/>
        <protocol type="TCPPING">
            <property name="initial_hosts">localhost[7600],localhost[7700]</property>
            <property name="port_range">10</property>
            <property name="timeout">3000</property>
            <property name="num_initial_members">2</property>
        </protocol>
        <protocol type="MERGE2"/>
        <protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>
        <protocol type="FD"/>
        <protocol type="VERIFY_SUSPECT"/>
        <protocol type="BARRIER"/>
        <protocol type="pbcast.NAKACK"/>
        <protocol type="UNICAST2"/>
        <protocol type="pbcast.STABLE"/>
        <protocol type="pbcast.GMS"/>
        <protocol type="UFC"/>
        <protocol type="MFC"/>
        <protocol type="FRAG2"/>
        <protocol type="RSVP"/>
    </stack>
</subsystem>

Then in your messaging subsystem comment out the socket-­binding section and include the jgroups-stack which references the JGroups stack (tcpping) and specify a jgroups­channel to be used by your cluster (if using different channels across your server groups you will end-up with a set of sub-clusters):

<broadcast­-groups>                         
   <broadcast-­group name="bg­group1">  
    <!--­­ <socket-­binding>messaging­group</socket-­binding> --­­>                                
   <jgroups­-stack>tcp</jgroups-­stack>                             
   <jgroups­-channel>hq­cluster1</jgroups-­channel>                                               
   <broadcast-­period>5000</broadcast-­period>                             
   <connector­-ref>netty</connector­-ref>                         
   </broadcast-­group>                     
</broadcast­-groups>                     

<discovery-­groups>                         
	<discovery­-group name="dg­group1">                      
        <!­­-- <socket­-binding>messaging­group</socket-­binding> --­­>   
	<jgroups-­stack>tcp</jgroups­-stack>                             
	<jgroups­-channel>hq­cluster1</jgroups­-channel>                             
	<refresh­-timeout>10000</refresh­-timeout>                         
	</discovery-­group>                     
</discovery-­groups>

 WildFly configuration

That above will work with JBoss EAP 6 or AS 7. If you are using WildFly replace the netty connector with the http-connector as shown here:

<broadcast-­groups>                         
   <broadcast-­group name="bg­group1">  
    <!-- ­­<socket-­binding>messaging­group</socket­-binding> --­­>                            
   <jgroups-­stack>tcpping</jgroups-­stack>                             
   <jgroups-­channel>hq­cluster1</jgroups-­channel>                                               
   <broadcast­-period>5000</broadcast-­period>                             
   <connector-­ref>http-connector</connector-­ref>                         
   </broadcast-­group>                     
</broadcast-­groups>                     

<discovery-­groups>                         
	<discovery-­group name="dg­group1">                      
	 <!­­-- <socket-­binding>messaging­group</socket-­binding> --­­>        
	<jgroups-­stack>tcpping</jgroups­-stack>                             
	<jgroups-­channel>hq­cluster1</jgroups­-channel>                             
	<refresh-­timeout>10000</refresh-­timeout>                         
	</discovery­group>                     
</discovery-­groups>

2) Using Clustered static discovery

Sometimes UDP is not available on your network so the initial discovery needs to be done using a static list of servers. This could be just one server that you know will always be available or a list of servers where at least one will be available.

In order to do that, perform the following steps:

1) Comment out the broadcast-group and discovery-groups at first:

<!--
	<broadcast-groups>
		<broadcast-group name="bg-group1">
			<socket-binding>messaging-group</socket-binding>
			<connector-ref>netty-connector</connector-ref>
		</broadcast-group>
	</broadcast-groups>
	<discovery-groups>
		<discovery-group name="dg-group1">
			<socket-binding>messaging-group</socket-binding>
		</discovery-group>
	</discovery-groups>  -->

 2) Next, within your cluster-connections section provide a static-connectors list:

<cluster-connections>
	<cluster-connection name="my-cluster">
		<connector-ref>netty-connector</connector-ref>
			<address>jms</address>                    
			<static-connectors> 
                             <connector-ref>cluster-node1</connector-ref>
			    <connector-ref>cluster-node2</connector-ref>
		        </static-connectors>
	</cluster-connection>	
</cluster-connections>

 3) The connectors  needs to be defined in the <connectors> section:

<connectors>
            <netty-connector name="cluster-node1" socket-binding="node1"/>
            <netty-connector name="cluster-node2" socket-binding="node2"/>
   . . . .
</connectors>

 4) The socket-binding in turn are defined into the socket-binding-group section as follows (supposing you have two other nodes on the same machine with an offset of 100 and 200):

<socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
. . . . .
	<outbound-socket-binding name="node1">
		<remote-destination host="localhost" port="8180"/>
	</outbound-socket-binding>		
	<outbound-socket-binding name="node2">
		<remote-destination host="localhost" port="8280"/>
	</outbound-socket-binding>						 
</socket-binding-group>

 WildFly static discover

The same configuration can be applied on WildFly: the only difference is that you will switch from a netty connector to an http-connector.

Step 1) Comment out the broadcast/discovery groups

<!--
	<broadcast-groups>
		<broadcast-group name="bg-group1">
			<socket-binding>messaging-group</socket-binding>
			<connector-ref>http-connector</connector-ref>
		</broadcast-group>
	</broadcast-groups>
	<discovery-groups>
		<discovery-group name="dg-group1">
			<socket-binding>messaging-group</socket-binding>
		</discovery-group>
	</discovery-groups>  -->

 Step 2) use an http-connector

<cluster-connections>
	<cluster-connection name="my-cluster">
		<connector-ref>http-connector</connector-ref>
			<address>jms</address>                    
			<static-connectors> 
                            <connector-ref>cluster-node1</connector-ref>
			    <connector-ref>cluster-node2</connector-ref>
		</static-connectors>
	</cluster-connection>	
</cluster-connections>

 Step 3) Define the Connectors:

<connectors>
      <http-connector name="cluster-node1" socket-binding="node1">
                        <param key="http-upgrade-endpoint" value="http-acceptor"/>
      </http-connector>	
      <http-connector name="cluster-node2" socket-binding="node2">
                        <param key="http-upgrade-endpoint" value="http-acceptor"/>
      </http-connector>
   . . . .
</connectors>

 Step 4) Define the socket bindings:

<socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
. . . . .
	<outbound-socket-binding name="node1">
		<remote-destination host="localhost" port="8180"/>
	</outbound-socket-binding>		
	<outbound-socket-binding name="node2">
		<remote-destination host="localhost" port="8280"/>
	</outbound-socket-binding>						 
</socket-binding-group>

 That’s all! Now your full-ha configuration will use a static list of servers.