BalaBit Blog

Continuous Integration and Testing Conference 2012 Europe took place in Budapest on the last weekend. Unlike other conferences, where a few people are selected to share their ideas and experiences with the crowd, it has an interesting format which I have never seen before, called OpenSpace. The event started in the evening of day #1 with a brief introduction on the topic, the conference, the format, the sponsors and such, then the participants introduced themselves to each other by answering three simple questions:

• Who are you?
• Where have you heard about the event?
• Why are you here?

Once we got a clue about each other, an empty agenda was shown hanging on the wall: participants were asked to fill it by proposing topics for discussions. Titles were written on post-its and were attached to the schedule somewhere while being introduced to the crowd. When the schedule got (over)filled, everybody grabbed a marker to vote on the topics and at the same time, the refactoring kicked off: similar discussions and titles got joined, topics were moved around between time slots and rooms based on the number of votes. You’d think that this lead to unusable chaos, but in fact, after a couple of minutes, the schedule evolved to a state where it started to stabilize.

However, anybody could change anything on the agenda at any time, talks could be switched even right before they’d have started, so it was recommended to check the schedule frequently: both for knowing the whats, wheres and whens, and for preventing conflicts between discussions you were planning to attend. For the rest of the event the most important rule was the Law of two feet which tells you to walk away from any discussion to another right when you realize that the topic is not exactly what you have expected. It’s not rude, it’s the expected behavior in that case! The evening ended with free chat in the hall with tasty snacks and some drinks.

For me, the second day started with a discussion on TDD at 10:00 AM. It was interesting to see how others use its principles and hear their stories about doing or not doing unit testing. We had developers who never write unit tests, some after-the-fact unit test writers, some TDD-ers, even BDD-ers as well. Since one of the initial questions was about finding personal motivation for learning and practising TDD, I added my five cents by recommending everyone to join or facilitate a Code Retreat near their locality, for example on 8th December, the Global Day of Code Retreat. We discussed a lot of aspects about unit testing, but the most valuable for me was the part about introducing TDD to fellow developers: most people won’t adopt methodologies like that until they realize they have a problem and then they see how it can be solved by the new methodology. And by the way: TDD does not automatically enforce better design, it’s experienced designer doing TDD who can do great designs! In the end, we agreed that TDD is like sex: once you have tried it, you never look back!

The second session I attended was about the after-life of builds: how do you get people to happily fix the builds they have just broken, what about deployment, how can you automatize it, how to upgrade or rollback data, etc. Right now I’m thinking about making our CI dashboard more visible, so for example, anyone entering our room would see our latest build statuses so we would be more motivated to keep it green and fix immediately. And by the way: do the impossible fifty times a day!

Next I took part in a discussion about making tests effective: we studied how and why the test pyramid often turns upside down, how the tests near the top get more and more slow and unstable, and how to fix them. These are so called second generation problems, since they imply that, at least, there is a bunch of automated tests for the code. There’s nothing wrong about running series of end-to-end or through-the-UI tests during whole nights frequently, but you need to care about the lower levels of your testing pyramid: the much faster unit tests (close to or above a thousand tests per second!) and integration tests. Have you heard about the Cup of Coffee Metric?

Oh, and it’s recommended to run a set of quick checks (learning tests if you like) before starting those whole-night testruns, just to see if the environment works as expected. Automated tests are code, so they should be treated like code as well: for example, decoupling from test frameworks can be a good idea. How old is your Selenium? Are you afraid of the cost of an upgrade? Are your tests cluttered with XPath expressions all over the place?

The fourth discussion I participated in was about knowledge transfer, especially regarding documentation. I like to take the “good code documents itself” advise literally, so I try to write the least possible amount of documentation. In certain cases, API docs is mandatory at least for public methods, so when writing those, I try to avoid quoting Captain Obvious. (You know, that guy is the one who writes a brief description for a method named writeStdOut() saying “write standard output”.) I name my tests so that even a non-techie person can understand the requirements I implemented from the verbose output of the test framework or from the list of tests run by the CI. But during this discussion, it turned out for me that in-code and runtime generated docs are not easily searchable for non-techies, and I cannot expect them for example to know that PHPUnit prints nice requirement list based on test method names when invoked with the --testdox commandline argument. I learned a great idea as well: in certain cases it’s a good idea to generate visualizations from tests code, for example, in case of a state transition diagram.

In the last time slot I visited the talk about risk analysis and risk management. We got ideas on thinking through the entire system to see what can fail, what parts of the system will a failure effect and how do we get to know about a failure. 5 whys vs. 5 so-whats.

The closing talk turned out to be a discussion as well: after some participants among those who filled out the feedback form won ebooks, the organizers, @Jtf and @PaulJulius asked everyone to share their biggest A’HA! moments. Man, I had plenty of such moments, especially during the TDD, the Effective testing and the Knowledge transfer sessions!

All in all, it was the most interesting, and perhaps the most valuable conference I’ve ever attended. Now it will take weeks or even months to process all the information, links and books that I learned about from the great variety of engineers I met there from so many different cultures, having so many different mindsets. Thank you CITCON, I hope to attend next year as well!

P.S: it’s pronounced kit-kon!