Here’s a tricky one; we need to do paging on one of our listing pages. The obvious solution was the following, which is a common pattern:

@Secured(value={Roles.PERSON_READ})
public List<Person> getPeople(int offset, int max) {
   Criteria criteria = getSession().createCriteria(Person.class);
   criteria.setFirstResult(offset;
   criteria.setMaxResults(max);
   return criteria.list();
}

Can you see it? The bug I mean? See it? How about now? Now?

The problem lies in the @Secured annotation. This little guy is SOOOOOO useful for us, but it will burn you once in a while. This annotation will proxy the method and remove any objects which you aren’t allowed to get back. i.e.: It will filter the returned list.

I guess you see it now: If I ask Hibernate to give me rows from 20 to 50, I am in no way garanteed to get back 30 results since Spring will remove items from my list AFTER the query has run and Hibernate has created the list.

The solution? I guess that really depends on your own requirements. Given our own volume and requirements, pulling out all the data on each page request is acceptable (we have a query cache enabled, so it’s not as bad as it may sound).

In other words, you want to test everything around the http call, but don’t care about the call actually being made or not. Yes, yes, yes. I know; just abstract out the call to another class, then mock that … pain in the ass! Use Spring rest template. Pain in the ass and overkill and doesn’t work well in multi threaded setups.

The solution we now use here is called NanoHTTPD which is a tiny, tiny, miniscule http server which can easily run inside a suite of tests. It can easily be extended to intercept calls. For example, we have:

• A HitCountingNanoHTTPD implementation which counts how many times the server was hit (good for testing that a multi-threaded class actually does the amount of call outs you need)
• Another implementation that given a file with key value pairs, will set the response headers using that file.

class HttpHitTest {
     private NanoHTTPD nano;

     @Before
     public void setup() throws Exception {
          nano = new Nano(9999);
     }

     @After
     public void tearDown() throws Exception {
          nano.stop();
     }

     @Test(expected=NoSuchFileException.class)
     public void testHttpHit() {
           HttpGetClass hgc = new HttpGetClass();
           hgc.doGet("http://localhost:9999/resources/myfile.mp3");
      }
}

See the beauty! NanoHTTPD simply exposes all files found in it’s working directory. So all test resources simply go into a folder inside the project and the CI server runs it without a hitch. Bonus: test performance and speeds are not affected (this thing is just so tiny).

As well, here’s what an extension to NanoHTTPD would look like. This example is a very simple something I whipped up to count how many times a server is hit (I needed the counter to be thread safe)

public class HitCountingNanoHttpd extends NanoHTTPD {
	private AtomicLong hits = new AtomicLong(0);

	public HitCountingNanoHttpd(int port) throws IOException {
		super(port);
	}

	@Override
	public Response serve(String uri, String method, Properties header, Properties parms, Properties files) {
		hits.incrementAndGet();
		return super.serve(uri, method, header, parms, files);
	}

	public long getHitCount() {
		return hits.longValue();
	}
}

My assessment just might not be as out of line as some may have thought. Good to get a bit of validation. I may sound like I’m gloating (I sort of am, because honestly some of what happened in the back streets of this story was a bit disgraceful …), but I’m not; I find it really unfortunate. I am still of those that badly wants ActiveMQ to make it.

Edit: Another follow up entry here.

First, let’s get this out of the way: I get what open source is; I get that there are a LOT of people who invest a LOT of their personal time; I get that while these people are working hard, I’m climbing cliffs instead of helping them even though I have the capacity and knowledge to do so; I get that criticism isn’t cool, especially when voiced publicly; and I get that I can be abrasive :).

With that said, in the last couple of months, we’ve worked hard to give ActiveMQ an nth chance. Unfortunately to no avail. The good folks at ActiveMQ have mentioned that we may be the first to use ActiveMQ the way we are. There has been a lot of effort into debugging the problem and there may be light at the end of the tunnel, but a fix is most likely a ways off.

The good news (for us that is), is that we have decided to keep ActiveMQ but in single broker configurations. Obviously, this is far from ideal, but all our products are currently programmed and architected around the possibility of a broker being unavailable, so we’ll keep on programming in the safeguards. I repeat: ActiveMQ is awesome in a single broker or embedded scenario.

The problem

The spring-data org.springframework.data.keyvalue.redis.connection.jedis.JedisConnectionFactory class offers no way to configure it to use a specific database. It defaults to database index 0 (in other words, does nothing on a new connection – so Redis automatically works on index 0).

Given that we will be running automated integration tests on a single Redis server instance, and that those tests will flush the database on each run (i.e.: select 2; flushdb), we needed an easy way to get a new connection factory instance that would first select the configured database index when a connection is created.

Without that database index selection, any integration tests running concurrently would flush or interfere with the data of the other running tests.

The solution

We simply extended the org.springframework.data.keyvalue.redis.connection.jedis.JedisConnectionFactory class and overrode the fetchJedisConnector() protected method:

public class MyCustomJedisConnectionPoolFactory extends JedisConnectionFactory {

private static final int DEFAULT_REDIS_DATABASE_INDEX = 0;

private int databaseIndex = 0;

@Override

protected Jedis fetchJedisConnector() {

Jedis jedis = super.fetchJedisConnector();

if (this.databaseIndex != DEFAULT_REDIS_DATABASE_INDEX) {

jedis.select(this.databaseIndex);

}

return jedis;

}

public void setDatabaseIndex(int databaseIndex) {

this.databaseIndex = databaseIndex;

}

}

The if statement isn’t strictly necessary in the fetchJedisConnector() method, but given that most of the time we will be using the default database, it seemed like it was worth it.

Word to the wise: Do NOT answer that you use the Java Date or Calendar classes. Too many times have women straight out told me that I was a looser with no self respect because of that answer.

I’m now at a point where I use exclusively Joda Time. And you should to. No bullshit formatting, date math and time zones. Add to that great documentation.

Non dynamic destination configuration through spring

<bean name="myDestination" class="org.hornetq.api.jms.HornetQJMSClient" factory-method="createQueue">
   <constructor-arg index="0" value="ExampleQueue"/>
</bean>
<bean name="jmsTemplate">
   <property name="connectionFactory" ref="connectionFactory"/>
   <property name="defaultDestination" ref="myDestination"/>
</bean>

In the above example, the queue TestQueue will already need to be defined on your HornetQ server.

Dynamic creation through a Spring FactoryBean implementation

Using a simple FactoryBean implementation and the HornetQ management api, we can achieve dynamic destination creation.

<bean name="myDestination" class="com.mikem.HornetQueueFactory">
   <property name="name" value="TestQueue"/>
</bean>
<bean name="jmsTemplate">
   <property name="connectionFactory" ref="connectionFactory"/>
   <property name="defaultDestination" ref="myDestination"/>
</bean>

Notice the different class used to define the myDestination bean. This is a simple implementation of a Spring FactoryBean.

 public class HornetQueueFactory implements FactoryBean {
	private String name;

	@Override
	public Class getObjectType() {
		return Queue.class;
	}

	@Override
	public boolean isSingleton() {
		return true;
	}

	@SuppressWarnings("unchecked")
	@Override
	public Queue getObject() throws Exception {
		boolean created = false;

        ObjectName on = ObjectNameBuilder.DEFAULT.getJMSServerObjectName();
        JMXConnector connector = JMXConnectorFactory.connect(new JMXServiceURL("service:jmx:rmi:///jndi/rmi://localhost:1097/jmxrmi"), new HashMap());
        MBeanServerConnection mbsc = connector.getMBeanServerConnection();
        JMSServerControl serverControl = (JMSServerControl)MBeanServerInvocationHandler.newProxyInstance(mbsc, on, JMSServerControl.class, false);

        String[] queueNames = serverControl.getQueueNames();
        if (!ArrayUtils.contains(queueNames, this.name)){
        	created = serverControl.createQueue(name);
        	System.out.println("Created new queue with name " + name);
        } else {
        	created = true;
        }

        if (!created) {
        	throw new RuntimeException("Queue isn't created. Not sure why.");
        }

        return HornetQJMSClient.createQueue(name);
	}

	@Required
	public void setName(String name) {
		this.name = name;
	}}

Notes:

• JMX/RMI needs to be configured and on for this to work
• Obviously, this is a naive implementation only used to see if it could be done.
• Once it’s all cleaned up, I will probably use a single factory for both Queues and Topics

When I started this job a couple years ago, we were mildly invested in ActiveMQ with a single queue for some mission critical work. Since I’d had good experience with ActiveMQ 4.x in previous jobs, I didn’t hesitate to push forward and suggest we invest more in ActiveMQ and architect solutions around it’s usage.  After all, I had experienced flawless performance and up time with 4.x.

Forward 2 years: We now have 30 or so topics and queues with moderate usage and it’s incapable of staying up more than a few minutes when configured as a network of brokers.

We’ve gone so far as paying the big bucks to get consulting from the folks at FuseSource. Unfortunately, the suggestions and help have been … uh … expensive with no results at all.

The setup

The setup is relatively straightforward:

• 3 brokers in different cities in different countries which need to work as a network of brokers.
• Latency can be high (80ms)
• Java clients using the client libraries over the tcp protocol
• C++ clients using the ActiveCPP library
• A few dozen topics and queues
• No big messages (5MB max with the average being closer to a few hundred k)
• Moderate traffic (a few hundred thousand messages per day overall)

Everything performs well enough in a single broker setup, but that’s un-acceptable for us.

What was done

Pretty much everything :)

Our expert had us upgrade to the latest version (5.4.2) with horrendous results; the KahaDB would continuously corrupt. Let me say this about trying to fix a basic problem with a minor upgrade: It’s bullshit. If version 5.4.1 can’t do something as straightforward as we’re trying to do, some suggestion about upgrading a point release isn’t going to do anything.(edit: The language was harsh. As it turns out, the minor upgrade was to adress another issue and not the network of brokers issue). And don’t be mistaken; our usage is basic and reasonable and definitely falls into the advertised set of features touted by ActiveMQ.

What’s next

We’re throwing all our weight behind JBoss HornetQ right now. The team is pretty psyched because they’re all fed up and disillusioned with the ActiveMQ product. Obviously, the migration path will not be easy, but at least we should no more be prisoner to a buggy product. (edit: Again, harsh: I’d say ‘prisoner to a product that doesn’t work in our configuration’)

Unit tests

Unit tests usually won’t cross the boundaries of a class. There are definitely times when it’s acceptable, but generally speaking, you would almost always inject any dependencies as mock objects.

Integration tests

Integration tests will take one (or more classes) inside the same component/functionality and test those together. At this point, I have no problem crossing the file system boundary, but I would hesitate to go to a database or send messages to a broker (for example)

Functional tests

Puts a whole system to the test and usually will use most of it’s external dependencies (DB, JMS, …). A functional test will check that the system and data flow is ok.

For example

At work, we’ve grouped together to work on a complex Camel route that includes many processors, daos, file system operations, etc … I’ve tasked myself with writing the message writing component, specifically the component that receives messages and writes them to disk (Yes! I’m aware that Camel has this, but ours needs to be much more custom).

My message writer looks something like this:

(Obviously, much more complex, but this will do for examples sake)

I have 2 unit test clases:

• One for the factory class. In the unit tests, the factory builds mock MessageWriter instances.
• Another for the MessageWriter class. This class has a FileSystemWriter instance injected so that I can actually get full test coverage. Without its, my unit tests would have crossed the file system boundary. This injected dependency allows me to replace it in unit tests with a mock.
• FileSystemWriter isn’t unit tested … This is left up to the integration tests

As for integration tests, we use Spring. So in another suite of JUnit tests, I use the actual Spring context with test properties, and run the whole component. Obviously, assertions are much more complicated as I need to go to the file system and read file content to ensure all is good.

This very loosely coupled system gives the team members a tremendous amount of confidence, flexibility and autonomy while developing their own components; they need only a smart interface describing the component, and they can develop in complete isolation.

Second, the atheist in me gets easily annoyed at a book that uses pastors and churches in examples …

But I’ll stick to it until the end :)