First August Meeting: GSoC 17 Parallel Testing
Date: 2nd August 2017
Time: 15:00 (UTC / GMT)
Meeting was attended by: Andrei Isac, Tito Alvarez, Niels Braczek, Astrid Guenther
1. Project Summary - work done so far and explanations
2. Discussed tasks vanishing problem and solved during meeting.
3. Work to do until next meeting
4. Real life use of the project discussion
1) Project Summary
The project is divided into three main parts (Virtualisation, Main coordinator and Execution Queue), and actually the last two can be seen as one.
The virtualisation part is done and working properly, generating the environment based on a configuration like this.
Then, the next problem was the selection list, which should load tasks from an yml file, managing them and their dependencies.
The Selection List is implemented here:
It has a recursive read function that reads the tests.yml file for weblinks, and a pop function that returns the next available task, checking that its dependencies are executed.
In the MCS class (short for MainCoordinatorStatic), everything is controlled(the execution flow and memcache manipulation)
I had to get a fast persistent storage because the tests should work async, therefore I have chosen memcached. Furthermore, I have developed a basic locking mechanism in order to avoid concurrency issues.
And now, the main flow is like this (for the whole run:coordinator task):
1. prepareExtension -> (clones the extension and generates the testing package)
2. generateEnv -> (generates the testing environment and starts the containers using the virtualisation package)
3. prepares the selection lists and other data stored in memcache
4. fillAndRun -> fills the execution queue and run the available task on the available clients;
When a task is run: the method run from the Task class is called:
It starts asynchronously a Robo Task that is in charge of running a docker exec command that runs the needed task on the needed clients for the needed server. In addition, it also manages the task, marking it assigned, failed or executed. So every time it is called, the fillAndRun method is called, which tries to fill the execution queue and run tasks if clients are available.
This should work totally asynchronously until all the tasks are executed or failed.
For now, the logs are stored in a file coordinator.log and the results can be observed there.
2) Tasks vanishing problem
There is one thing I have not managed to figure out, after a few tasks run, it seems that some processes that I start just vanish. The result is that the clients are blocked, because a task has been assigned for them, but the actual test is never run, practically blocking the whole workflow, because all the clients get blocked at some point.
In order to understand the flow:
so, the expected workflow for a task is:
1. run the task liek this: $task->run(); -> this will start a process and start() it, which does not wait for it to finish.
2. The robo task(runClientTask) is called;
3. Task is marked as assigned and the queue is filled again and task are run if clients and tasks are available(this is the fillAndRun behaviour);
4. docker exec command is run and $result is checked for task success;
5. if ok then assign as executed and fillAndRun
6. if not then assign as failed and fillAndRun
Everything works fine, until one process that calls runClientTask is started, but does nothing; The result is that it blocks the client.
So after enough processes "vanish" all the clients are blocked.
I have solved the problem. It was caused by a Symfony process issue/feature, because I was using $process->start() which does not wait for output, but I was already redirecting all output to a log file. So I have changed it to $process->run(). Now it works fine for one server and multiple clients.
I shall test it for multiple servers and multiple clients. Problems may arise here because the acceptance.suite.yml is reused and there may be concurrency issues.
Until next week is to finish testing the current setup, improve the code quality, as now everything is inside the MCS class and write comments for each function.
4) Real life application
1.How will this project be integrated in real life?
The idea is that each Joomla extension can integrate this project via composer, to have parallel test results run via Travis or Jenkins (any CI) and report back to Github
2. I know I may use the reporting tool from joomla-testing-robo, but how exactly is reporting desired?
We never specified reporting in reality, but for the scope of this project, each testing script can report back their own errors directly in Github comments. This is to be improved after the current project, because the best way to handle this is to centralize reporting in the main coordinator, but that part is not that critical as the parallel execution is, so that's why we decided to leave it out of scope
The reporting data that would be needed is:
1. The full selenium and codeception log with all the execution (verbose)
2. The tap log of the scenario returning with error
3. The image of the scenario with the error