mikem
This user hasn't shared any biographical information
Jabber/GTalk: mikem
Posts by mikem
Hibernate, Spring Security and list paging
Nov 22nd
Of the dangers of using Spring Security, Hibernate and paging
Here’s a tricky one; we need to do paging on one of our listing pages. The obvious solution was the following, which is a common pattern:
@Secured(value={Roles.PERSON_READ})
public List<Person> getPeople(int offset, int max) {
Criteria criteria = getSession().createCriteria(Person.class);
criteria.setFirstResult(offset;
criteria.setMaxResults(max);
return criteria.list();
}
Can you see it? The bug I mean? See it? How about now? Now?
The problem lies in the @Secured annotation. This little guy is SOOOOOO useful for us, but it will burn you once in a while. This annotation will proxy the method and remove any objects which you aren’t allowed to get back. i.e.: It will filter the returned list.
I guess you see it now: If I ask Hibernate to give me rows from 20 to 50, I am in no way garanteed to get back 30 results since Spring will remove items from my list AFTER the query has run and Hibernate has created the list.
The solution? I guess that really depends on your own requirements. Given our own volume and requirements, pulling out all the data on each page request is acceptable (we have a query cache enabled, so it’s not as bad as it may sound).
Testing http calls with NanoHTTPD
Aug 2nd
Well, we finally got tired of over engineering 2 lines of code just to make it testable. I’m talking about those classes which have some http calls done throughout some code; intermingled with maybe some error handling, some conditional logic, etc ..
In other words, you want to test everything around the http call, but don’t care about the call actually being made or not. Yes, yes, yes. I know; just abstract out the call to another class, then mock that … pain in the ass! Use Spring rest template. Pain in the ass and overkill and doesn’t work well in multi threaded setups.
The solution we now use here is called NanoHTTPD which is a tiny, tiny, miniscule http server which can easily run inside a suite of tests. It can easily be extended to intercept calls. For example, we have:
- A HitCountingNanoHTTPD implementation which counts how many times the server was hit (good for testing that a multi-threaded class actually does the amount of call outs you need)
- Another implementation that given a file with key value pairs, will set the response headers using that file.
class HttpHitTest {
private NanoHTTPD nano;
@Before
public void setup() throws Exception {
nano = new Nano(9999);
}
@After
public void tearDown() throws Exception {
nano.stop();
}
@Test(expected=NoSuchFileException.class)
public void testHttpHit() {
HttpGetClass hgc = new HttpGetClass();
hgc.doGet("http://localhost:9999/resources/myfile.mp3");
}
}
See the beauty! NanoHTTPD simply exposes all files found in it’s working directory. So all test resources simply go into a folder inside the project and the CI server runs it without a hitch. Bonus: test performance and speeds are not affected (this thing is just so tiny).
As well, here’s what an extension to NanoHTTPD would look like. This example is a very simple something I whipped up to count how many times a server is hit (I needed the counter to be thread safe)
public class HitCountingNanoHttpd extends NanoHTTPD {
private AtomicLong hits = new AtomicLong(0);
public HitCountingNanoHttpd(int port) throws IOException {
super(port);
}
@Override
public Response serve(String uri, String method, Properties header, Properties parms, Properties files) {
hits.incrementAndGet();
return super.serve(uri, method, header, parms, files);
}
public long getHitCount() {
return hits.longValue();
}
}
ActiveMQ vs. PrimeTime
May 26th
Well … what have we here.
My assessment just might not be as out of line as some may have thought. Good to get a bit of validation. I may sound like I’m gloating (I sort of am, because honestly some of what happened in the back streets of this story was a bit disgraceful …), but I’m not; I find it really unfortunate. I am still of those that badly wants ActiveMQ to make it.
Edit: Another follow up entry here.
The easy road
Mar 7th
To doubt everything or to believe everything are two equally convenient solutions; both dispense with the necessity of reflection.
H. Pointcaré
ActiveMQ issues update
Jan 27th
This is a follow up to a previous entry I made about problems experienced with ActiveMQ 5.
First, let’s get this out of the way: I get what open source is; I get that there are a LOT of people who invest a LOT of their personal time; I get that while these people are working hard, I’m climbing cliffs instead of helping them even though I have the capacity and knowledge to do so; I get that criticism isn’t cool, especially when voiced publicly; and I get that I can be abrasive :).
With that said, in the last couple of months, we’ve worked hard to give ActiveMQ an nth chance. Unfortunately to no avail. The good folks at ActiveMQ have mentioned that we may be the first to use ActiveMQ the way we are. There has been a lot of effort into debugging the problem and there may be light at the end of the tunnel, but a fix is most likely a ways off.
The good news (for us that is), is that we have decided to keep ActiveMQ but in single broker configurations. Obviously, this is far from ideal, but all our products are currently programmed and architected around the possibility of a broker being unavailable, so we’ll keep on programming in the safeguards. I repeat: ActiveMQ is awesome in a single broker or embedded scenario.
Extending Spring-Data to select a database index with Jedis pools
Jan 16th
We came up with a not so un-common use-case of wanting to setup a Jedis connection pool that would automatically select a given database index on each connection. A colleague and I were about to patch the Spring-Data source code (we are already running a patched version), but instead found a very elegant solution by digging in a bit more:
The problem
The spring-data org.springframework.data.keyvalue.redis.connection.jedis.JedisConnectionFactory class offers no way to configure it to use a specific database. It defaults to database index 0 (in other words, does nothing on a new connection – so Redis automatically works on index 0).
Given that we will be running automated integration tests on a single Redis server instance, and that those tests will flush the database on each run (i.e.: select 2; flushdb), we needed an easy way to get a new connection factory instance that would first select the configured database index when a connection is created.
Without that database index selection, any integration tests running concurrently would flush or interfere with the data of the other running tests.
The solution
We simply extended the org.springframework.data.keyvalue.redis.connection.jedis.JedisConnectionFactory class and overrode the fetchJedisConnector() protected method:
public class MyCustomJedisConnectionPoolFactory extends JedisConnectionFactory {
private static final int DEFAULT_REDIS_DATABASE_INDEX = 0;
private int databaseIndex = 0;
@Override
protected Jedis fetchJedisConnector() {
Jedis jedis = super.fetchJedisConnector();
if (this.databaseIndex != DEFAULT_REDIS_DATABASE_INDEX) {
jedis.select(this.databaseIndex);
}
return jedis;
}
public void setDatabaseIndex(int databaseIndex) {
this.databaseIndex = databaseIndex;
}
}
The if statement isn’t strictly necessary in the fetchJedisConnector() method, but given that most of the time we will be using the default database, it seemed like it was worth it.
Joda Time if you appreciate a good quality of life
Dec 7th
I’ve had it! When I go out clubbing and hit on women, one of the things that invariably comes up in the conversation is which date library I code to when using my made skillz in Java.
Word to the wise: Do NOT answer that you use the Java Date or Calendar classes. Too many times have women straight out told me that I was a looser with no self respect because of that answer.
I’m now at a point where I use exclusively Joda Time. And you should to. No bullshit formatting, date math and time zones. Add to that great documentation.
HornetQ dynamic destination creation and references through Spring
Dec 7th
As per my last post, we are currently looking into migrating from ActiveMQ towards HornetQ. One of the missing features which we used and depend on are dynamic queues and topics. Our main goal is to avoid touching any java code and depend solely on Spring configuration changes.
Non dynamic destination configuration through spring
<bean name="myDestination" class="org.hornetq.api.jms.HornetQJMSClient" factory-method="createQueue"> <constructor-arg index="0" value="ExampleQueue"/> </bean> <bean name="jmsTemplate"> <property name="connectionFactory" ref="connectionFactory"/> <property name="defaultDestination" ref="myDestination"/> </bean>
In the above example, the queue TestQueue will already need to be defined on your HornetQ server.
Dynamic creation through a Spring FactoryBean implementation
Using a simple FactoryBean implementation and the HornetQ management api, we can achieve dynamic destination creation.
<bean name="myDestination" class="com.mikem.HornetQueueFactory"> <property name="name" value="TestQueue"/> </bean> <bean name="jmsTemplate"> <property name="connectionFactory" ref="connectionFactory"/> <property name="defaultDestination" ref="myDestination"/> </bean>
Notice the different class used to define the myDestination bean. This is a simple implementation of a Spring FactoryBean.
public class HornetQueueFactory implements FactoryBean {
private String name;
@Override
public Class getObjectType() {
return Queue.class;
}
@Override
public boolean isSingleton() {
return true;
}
@SuppressWarnings("unchecked")
@Override
public Queue getObject() throws Exception {
boolean created = false;
ObjectName on = ObjectNameBuilder.DEFAULT.getJMSServerObjectName();
JMXConnector connector = JMXConnectorFactory.connect(new JMXServiceURL("service:jmx:rmi:///jndi/rmi://localhost:1097/jmxrmi"), new HashMap());
MBeanServerConnection mbsc = connector.getMBeanServerConnection();
JMSServerControl serverControl = (JMSServerControl)MBeanServerInvocationHandler.newProxyInstance(mbsc, on, JMSServerControl.class, false);
String[] queueNames = serverControl.getQueueNames();
if (!ArrayUtils.contains(queueNames, this.name)){
created = serverControl.createQueue(name);
System.out.println("Created new queue with name " + name);
} else {
created = true;
}
if (!created) {
throw new RuntimeException("Queue isn't created. Not sure why.");
}
return HornetQJMSClient.createQueue(name);
}
@Required
public void setName(String name) {
this.name = name;
}}
Notes:
- JMX/RMI needs to be configured and on for this to work
- Obviously, this is a naive implementation only used to see if it could be done.
- Once it’s all cleaned up, I will probably use a single factory for both Queues and Topics
ActiveMQ – not what it used to be?
Dec 7th
I’ve posted a quick update to this here.
When I started this job a couple years ago, we were mildly invested in ActiveMQ with a single queue for some mission critical work. Since I’d had good experience with ActiveMQ 4.x in previous jobs, I didn’t hesitate to push forward and suggest we invest more in ActiveMQ and architect solutions around it’s usage. After all, I had experienced flawless performance and up time with 4.x.
Forward 2 years: We now have 30 or so topics and queues with moderate usage and it’s incapable of staying up more than a few minutes when configured as a network of brokers.
We’ve gone so far as paying the big bucks to get consulting from the folks at FuseSource. Unfortunately, the suggestions and help have been … uh … expensive with no results at all.
The setup
The setup is relatively straightforward:
- 3 brokers in different cities in different countries which need to work as a network of brokers.
- Latency can be high (80ms)
- Java clients using the client libraries over the tcp protocol
- C++ clients using the ActiveCPP library
- A few dozen topics and queues
- No big messages (5MB max with the average being closer to a few hundred k)
- Moderate traffic (a few hundred thousand messages per day overall)
Everything performs well enough in a single broker setup, but that’s un-acceptable for us.
What was done
Pretty much everything :)
Our expert had us upgrade to the latest version (5.4.2) with horrendous results; the KahaDB would continuously corrupt. Let me say this about trying to fix a basic problem with a minor upgrade: It’s bullshit. If version 5.4.1 can’t do something as straightforward as we’re trying to do, some suggestion about upgrading a point release isn’t going to do anything.(edit: The language was harsh. As it turns out, the minor upgrade was to adress another issue and not the network of brokers issue). And don’t be mistaken; our usage is basic and reasonable and definitely falls into the advertised set of features touted by ActiveMQ.
What’s next
We’re throwing all our weight behind JBoss HornetQ right now. The team is pretty psyched because they’re all fed up and disillusioned with the ActiveMQ product. Obviously, the migration path will not be easy, but at least we should no more be prisoner to a buggy product. (edit: Again, harsh: I’d say ‘prisoner to a product that doesn’t work in our configuration’)
Unit vs. Integration vs. Functional testing
Nov 6th
In a recent conversation with a colleague we discussed where the line was drawn between types of tests.
Unit tests
Unit tests usually won’t cross the boundaries of a class. There are definitely times when it’s acceptable, but generally speaking, you would almost always inject any dependencies as mock objects.
Integration tests
Integration tests will take one (or more classes) inside the same component/functionality and test those together. At this point, I have no problem crossing the file system boundary, but I would hesitate to go to a database or send messages to a broker (for example)
Functional tests
Puts a whole system to the test and usually will use most of it’s external dependencies (DB, JMS, …). A functional test will check that the system and data flow is ok.
For example
At work, we’ve grouped together to work on a complex Camel route that includes many processors, daos, file system operations, etc … I’ve tasked myself with writing the message writing component, specifically the component that receives messages and writes them to disk (Yes! I’m aware that Camel has this, but ours needs to be much more custom).
My message writer looks something like this:
(Obviously, much more complex, but this will do for examples sake)
I have 2 unit test clases:
- One for the factory class. In the unit tests, the factory builds mock MessageWriter instances.
- Another for the MessageWriter class. This class has a FileSystemWriter instance injected so that I can actually get full test coverage. Without its, my unit tests would have crossed the file system boundary. This injected dependency allows me to replace it in unit tests with a mock.
- FileSystemWriter isn’t unit tested … This is left up to the integration tests
As for integration tests, we use Spring. So in another suite of JUnit tests, I use the actual Spring context with test properties, and run the whole component. Obviously, assertions are much more complicated as I need to go to the file system and read file content to ensure all is good.
This very loosely coupled system gives the team members a tremendous amount of confidence, flexibility and autonomy while developing their own components; they need only a smart interface describing the component, and they can develop in complete isolation.