By Hannes Papenberg on 2019-10-16 20:04 in Automated Testing Team

Last weekend, from October 11th to the 13th, the Joomla Automated Testing Team met in Essen in the Unperfekthaus. The Automated Testing Team is managing the continous integration setup of the Joomla project, which runs checks for each and every code change that Joomla accepts. These checks consist of checks for codestyle and a security check, as well as running a suite of automated tests to check that everything still works as expected. With the upcoming Pizza, Bugs and Fun, we had to especially concentrate on fixing issues we had with unstable tests.
At the sprint we were five team members (David Jardin, Hannes Papenberg, Jelle Kok, Puneet Kala and Robert Deutz) and this is what we achieved in those three days:

Updates & Performance

As a first step, we updated our Drone CI system to the current 1.6.0 version and in a rather large update, switched over our RIPS setup (a security scanner for PHP) to the latest version. With that out of the way, we then could look at adding more power to our CI setup. With the support of Hosting.de and Rochen.com, we were finally able to extend our servers from just one machine for the CI system to a multi-server setup. When before we could only run two jobs in parallel, we are now able to execute 15 at a time, thus greatly improving the wait times for developers. This has been about half a year in the making and I am very happy that this has been possible in time for PBF19. All those servers are also now covered by a monitoring system, alerting us right away when there are any issues. Last but not least, we did some optimisations in the order of execution of steps, which should save about half a minute in the total runtime of our CI system.

Simplifications & Cleanup

As wíth most of the code in Joomla, our tests as well have evolved over time and that means, that from time to time a review of our setup can result in awesome simplifications. While reviewing our system tests, we started to question our use of Joomla Robo. Joomla Robo is meant to be a tool for CLI actions, like setting up the test installation, starting Selenium, etc. Unfortunately the current implementation was not complete and introduced quite some complexity. Our old setup meant, that Drone fired up a Docker image, which fired up a shell script, which started Apache and Chrome and then Joomla Robo, which copied the test installation, started Selenium and then called Codeception, which started the tests, which started PHPUnit and then executed the actual tests... Especially since the Joomla Robo implementation was almost 500 lines of code. So we looked at ways to reduce that complexity and decided to drop Joomla Robo. The tasks of Joomla Robo could be replaced by extending our shell script by 10 lines of code. All that should make it a lot easier especially for people who are new to our tests. One sideeffect was, that we saved about a minute on our system tests for each run. Since we are running that setup both for MySQL, MySQL8, PostgreSQL and our API tests, that means we were able to reduce that runtime by 4 minutes.
Besides dropping Joomla Robo and the related dependencies, we also dropped our self-maintained version of Selenium and are now using Selenium from npm. This should make running these system tests on a local machine a lot easier and keep us rather up-to-date with the Selenium versions, while also saving several MB of disk space per J4 installation. As a last step, we also reorganised and updated all our docker images and in the future you will find all of those in https://github.com/joomla-projects/docker-images These will mainly be images for Joomla 4.x.

Bugfixing

Especially in the last few months we had serious issues with random fails of our tests. Any easy fix for failures in npm was, to switch off the caching we were doing in Drone. Turns out: Storing and retrieving that cache takes about as long as simply installing everything from scratch each time... But the biggest problem were race conditions in the system tests. We had several cases, where Selenium tried to click on elements which were still being setup or in animation and thus it was pretty much a question of chance of the tests passing or failing. These issues are really hard to debug and we are really happy that we apparently finally found all of the race conditions in our current tests. As far as we can tell, with these changes, we really only have legitimate test fails. No more running the tests 5 times to get one successfull pass.

And there is more

Besides all this work, we were also able to look a bit into the future. We experimented with CodeceptJS and are still evaluating if we should switch over with our system tests to this. We also are now at a point, where we can extend our testsuites in general and I'm looking forward to raise the coverage of Joomla 4. Thanks to Puneet for starting with additional system tests already. But most importantly we discussed the future of the Patchtester component.
The Patchtester is a great component to test pull requests without having to setup a full development environment and is an important part of the PBF19. With Joomla 4 we do however run into some issues, mainly to properly compile JS and CSS assets, which Joomla 4 generally builds from scratch out of the source files and their dependencies. We found a way to provide these files from our CI system and the Patchtester was adapted to work with this new datasource. In the past it was maintained by Michael Babker (thank you very much for that!) but didn't have an official team. In the future a workgroup is supposed to maintain it and since the file packages are created by the CI system, we would create this under the umbrella of the ATT, at least for now.

I want to thank all who participated in this sprint and would overall call it a pretty big success. With this event, I think we are setup pretty good to support Joomla 4 and its development and I'm looking forward to what we can achieve now.

Regards,
Hannes Papenberg
ATT Lead