How to use distributed JMeter to test netty server performance

JMeter is good at performance testing. Usually, we often need more than one hosts to send traffic to one server to see how the sever behaves. This Many-To-One test scenario is called distributed testing.  In JMeter's world, the hosts, which generate traffic,  are called 'jmeter server'. You can think them as slaves. The one, on which you setup test plan and control the start/stop of others, is called 'jmeter client' . You can think this only one as master.

Netty is a high performanc NIO framework. In this demo, a simple Echo server written with Netty 4 is what's going to be tested.  This demo use 5 other hosts to generate traffic to the echo server with long connection.  Everyone establishes 20,000 connections to the server and 100,000 connections totally for 5 hosts.

This article wants to bring you:

1. Basic concepts of JMeter. like Thread group, Logic Controller, Sampler, Timer and Listener,
2. How to setup a jmeter testplan with loop, thread group, timer, sampler and listeners
3. How to run jmeter in distributed test
4. How to read the test result.
5. Have a basic understanding of netty's performance. Since only 5 hosts used to generate traffic,  we can't fully show the capacity of the server, but only some idea of the workload in 100,000 connections. 

The JMeter version used in this article is 2.13. The server under test works as an Echo server, just send back what is recieved. The Echo server's code can be found in the end of this article.

1. JMeter basic elements

Every test you run is called a 'test plan', it's a container of other elements.  In a 'test plan', there is usually a 'thread group', which defines the concurrency of your test, like how many thread you want to use to run the 'sampler'.

What is 'sampler'? it's kinda of a real action the test performs, like send out a http request, a tcp request or make a JDBC connection.  'Timer' can help your test wait for some time before perform another action. The name of 'Listener' is a little abstract, but what 'listener' do is just show you the result of the test.  You can have more than one listeners to show the result in different format. e.g. in table or in chart.

JMeter also bunch of 'logic controller' to help you control test logic. In this demo, loop is used to send TCP request repeatedly.

2.Config JMeter test plan.

Test plan is the running unit of JMeter. It has all the test logic defined in it. In distributed testing, this test plan only need to be defined once on the master, and will be submitted to all slaves hosts automatically.

Let's define a test plan to send tcp request repeatly.  The steps are common for other kind of tests.

2.1 Add 'Thread Group'

Right click the test plan and add a thread group.

The default thread group only have 1 thread. Change it to 20,000.

2.2 Add Loop

we need a thread send tcp requst, recieve the response, then wait for a few seconds and send again. So we need a loop. In this loop we send tcp request and sleep.  Right click the thread group, add loop controller.

For this demo, let's choose loop forever.

2.3 Add Sampler

Now we have a big goup of thread ready to do something. Sampler defines what the real task is.  Right click on the loop controller, add a tcp sampler.

The tcp sampler needs some configuration. 

Besides the ip and port of the server, there are 3 places need to config. First re-use connection for 'long connection'. Secondly set String 'send message\n'  as the payload sent to server. Finally, set the EOL byte, jmeter default is 0x00, since we send string message, '\n' is the delimer for every message.

2.4 Add timer

Add a wait after each tcp request. Right click TCP sampler, add a constant timer.

Change the delay to 5000 ms.

Now the test logic is done, but we need to add listerner to check the test result.

2.5 Add listeners

We add 2 listeners by right click on the test plan.  One is 'Aggregate Report' to see the statistic result of test. The other listener dispaly response time in line chart. (It's a jmeter plugin from the plugin standard set, see http://jmeter-plugins.org/) . The test plan is ready to run, now it looks like below. 

3. Run the test locally

Select 'Run' ->'Start', let's run the test locally.  We can see the test result from listeners. The 'Aggregate Report' listener looks like below.

The 'Response Times Over Time' listener has the response time as a line chart.

So far so good. The sever seems can handle client from 1 host very easily. Next we are going to add more host to generate traffice. This's so-called distributed testing.

4. Run the test remotely

4.1 Prerequisition

• Every slave hosts has same version jmeter install. (ver 2.13 in this demo)
• all host in the same subnet to reduce the network delay.

4.2 Change properties file of 'master' jmeter

Modify ${JMeter_Install_Dir}/bin/jmeter.properties file, add all slaves' ip

Also in this file, change jmeter mode to 'StrippedAsynch' to reduce master's workload. This's necessary when play with large concurrency performance test.

Remember only change this on the master's host.  Now if start the jmeter on master host, you can see the slaves from menu.

Now the master, or we can say jmeter client,  is ready. 

4.3 Start JMeter engine on every slave host

Run 'jmeter-server' command on every slave hosts.

4.4 Run remotely

On master, click  run remote all button.

Very soon you will find the active thread goes up to 100k (20k x 5 slaves)

Now the test result looks like below.

Check  the server's workload, cpu usage is still quite low. On our 2CPU server ( Intel Xeon E5-27750 2.0G 20M Cache, total 32 thread cores), only about 2 cores are used. 

Although the echo server has no real logic, the nice performance of Netty framework is impressive.

5. More

Here are the source of our Netty 4 echo server.  It has 2 classes. First the ServerHandler.java

package com.shengwang.demo;

import io.netty.channel.ChannelHandlerContext;
import io.netty.channel.ChannelInboundHandlerAdapter;

public class ServerHandler extends ChannelInboundHandlerAdapter {

  @Override
  public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
    ctx.writeAndFlush(msg);  // loop message back
  }
}

Then the main class.

package com.shengwang.demo;

import io.netty.bootstrap.ServerBootstrap;
import io.netty.channel.ChannelInitializer;
import io.netty.channel.ChannelOption;
import io.netty.channel.ChannelPipeline;
import io.netty.channel.nio.NioEventLoopGroup;
import io.netty.channel.socket.SocketChannel;
import io.netty.channel.socket.nio.NioServerSocketChannel;
import io.netty.handler.codec.string.StringDecoder;
import io.netty.handler.codec.string.StringEncoder;
import io.netty.util.concurrent.DefaultEventExecutorGroup;
import io.netty.util.concurrent.EventExecutorGroup;

import java.io.IOException;

public class NettyServer {

  public static void main(String[] args) throws IOException, InterruptedException {
    NioEventLoopGroup boosGroup = new NioEventLoopGroup();
    NioEventLoopGroup workerGroup = new NioEventLoopGroup();
    ServerBootstrap bootstrap = new ServerBootstrap();
    bootstrap.group(boosGroup, workerGroup);
    bootstrap.channel(NioServerSocketChannel.class);
    
    // ===========================================================
    // 1. define a separate thread pool to execute handlers with
    // slow business logic. e.g database operation
    // ===========================================================
    final EventExecutorGroup group = new DefaultEventExecutorGroup(1500); //thread pool of 500
    
    bootstrap.childHandler(new ChannelInitializer<SocketChannel>() {
      @Override
      protected void initChannel(SocketChannel ch) throws Exception {
        ChannelPipeline pipeline = ch.pipeline();
        pipeline.addLast(new StringEncoder()); 
        pipeline.addLast(new StringDecoder()); 
        
        //===========================================================
        // 2. run handler with slow business logic 
        //    in separate thread from I/O thread
        //===========================================================
        pipeline.addLast(group,"serverHandler",new ServerHandler()); 
      }
    });
    
    bootstrap.childOption(ChannelOption.SO_KEEPALIVE, true);
    bootstrap.bind(19000).sync();
  }
}

This echo server depends on netty 4.0.34.

    <dependency>
      <groupId>io.netty</groupId>
      <artifactId>netty-all</artifactId>
      <version>4.0.34.Final</version>
    </dependency>