Performance issue running coverage on Windows #1062
Comments
sebastianbergmann
changed the title
|
There have been no changes (that I am aware of) that are specific to branch/path coverage. From your Before I go down the wrong rabbit hole, can you please check whether you also get the "hanging" when branch/path coverage is disabled? |
|
@staabm Can you have a look at https://github.com/sebastianbergmann/php-code-coverage/blob/main/src/Target/MapBuilder.php and see if/how this could be optimized? Thanks! |
|
I can have a look after Juliette pushed the phpunit 12 changes for the reproducer |
I think any project can be used to look into the performance of |
|
I was not yet able to reproduce the slow run on https://github.com/Yoast/PHPUnit-Polyfills (I have no experience with branch coverage): do you think we can trigger the slow path without xdebug beeing loaded (e.g. with pre-recorded information)? running blackfire and xdebug in the same process usually doesn't work, as these step onto each others feet. |
|
I think we should wait until Juliette answered my "can you please check whether you also get the "hanging" when branch/path coverage is disabled?" question. I am almost certain that this is not related to branch/path coverage. |
@sebastianbergmann I've just checked and with only line coverage and filtered down to one test class, I see a small, but not a significant slow down. LogsWhen running the full suite with only line coverage, I see no significant difference in the performance. LogsSo, for argument's sake, I checked the performance of path/branch coverage when running the full test suite as well, and the slow down is back, so that confirms it's the path/branch coverage which is the source of the slow-down, not the test filtering. Running the full test suite vs a filtered down test doesn't seem to make a difference in the amount of slow down though. Logs
@staabm The WIP branch I'm using is available, though not yet merged: https://github.com/Yoast/PHPUnit-Polyfills/commits/feature/composer-ci-allow-for-phpunit-12/ Just FYI/I'll figure out some hacky solution - The thing I'm running into is something which was fixed in PHPUnit 11.3.? (non-existent traits not being ignored, but causing a fatal). As this package polyfills a PHPUnit 11.2.0 assertion ( I suspect I'll end up creating PHPUnit version specific wrappers for the tests + adding another config file. Nothing to worry about though as this is an edge-case which should not affect "normal" packages. |
@staabm In case it helps, this is the exact config I used locally: <?xml version="1.0" encoding="UTF-8"?>
<phpunit
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="https://schema.phpunit.de/10.5/phpunit.xsd"
backupGlobals="true"
bootstrap="./tests/bootstrap.php"
beStrictAboutOutputDuringTests="true"
beStrictAboutTestsThatDoNotTestAnything="true"
colors="true"
cacheDirectory=".cache/phpunit.cache"
requireCoverageMetadata="true"
displayDetailsOnTestsThatTriggerErrors="true"
displayDetailsOnTestsThatTriggerWarnings="true"
displayDetailsOnTestsThatTriggerNotices="true"
displayDetailsOnTestsThatTriggerDeprecations="true"
displayDetailsOnPhpunitDeprecations="true"
failOnWarning="true"
failOnNotice="true"
failOnDeprecation="true"
failOnPhpunitDeprecation="false"
>
<testsuites>
<testsuite name="PHPUnitPolyfills">
<directory>./tests/</directory>
</testsuite>
</testsuites>
<source>
<include>
<file>phpunitpolyfills-autoload.php</file>
<directory suffix=".php">./src/</directory>
</include>
<exclude>
<file>src/Polyfills/AssertClosedResource_Empty.php</file>
</exclude>
</source>
<coverage pathCoverage="true">
<report>
<text outputFile="php://stdout" showOnlySummary="true"/>
<html outputDirectory="build/coverage-html" lowUpperBound="35" highLowerBound="90"/>
</report>
</coverage>
</phpunit> |
|
which I am still unable to reproduce with using |
@staabm This the config I'm using for PHP 8.4.1 (which I used for the above tests): [xdebug]
zend_extension="C:/wamp/bin/php/ext/xdebug/php_xdebug-3.4.1-8.4-ts-vs17-x86_64.dll"
;xdebug.mode allowed are : off develop coverage debug gcstats profile trace
xdebug.mode=coverage
xdebug.output_dir="C:/wamp/tmp"
xdebug.show_local_vars=0
xdebug.log="C:/wamp/logs/xdebug.log"
;xdebug.log_level : 0 Criticals, 1 Connection, 3 Warnings, 5 Communication, 7 Information, 10 Debug Breakpoint
xdebug.log_level=7
xdebug.profiler_output_name=trace.%H.%t.%p.cgrind
xdebug.use_compression=false |
|
For argument's sake just reran the tests with PHP 8.3.14 (same Xdebug config) and got the same results. |
|
@staabm Another thought, though I can't imagine it would make a difference - I'm running the tests with the PHAR file, while it looks l like you're trying to reproduce with a Composer installed version. Update: just tested and to my dismay and surprise - I cannot reproduce with a Composer install either. So the PHAR in combination with path/branch coverage is where things are going wrong... Colour me flabbergasted. |
|
There seems to be something broken with regard to the static analysis cache when the PHAR is used. Although different, sebastianbergmann/phpunit#6124 seems to point into that direction as well. |
|
@jrfnl Can you delete your |
Done (including deleting various other files, like PHPUnit 9 cache file and code coverage result files) and it unfortunately doesn't make a difference. |
|
Please test whether the PHPUnit 12.0.2 PHAR solves this issue for you. Thank you! |
|
@sebastianbergmann Tested and unfortunately, no luck. |
|
Thank you for testing. I will (try to) stop working on this until after the weekend. Besides, I have no idea what might be going on here as I looked into the changes between In general, I can only advise against using branch/path coverage "all the time" (as in configured in your XML configuration file). But, of course, that is not a solution to the problem you discovered. |
|
@sebastianbergmann Sounds like a good plan. Wishing you a relaxing weekend! Let me know if/when you want me to try additional debug steps. Happy to oblige.
Agreed. The default config for projects I manage is generally set to line coverage. It's only my local override config in which I have path/branch coverage enabled and even then, I normally only run coverage locally when making significant changes. Or, like in this case, when debugging an issue with code coverage ;-) |
The only thing I can think of right now would be to use git-bisect to find out which commit in |
|
🤔 I seem to remember setting up making PHARs available as artifacts for the builds in the PHPUnit repo. I might be able to work something out from that. |
|
|
|
Good to know, though I've never managed to get |
|
Yeah, that's not going to work. I've managed to get |
|
Sorry, but you need something UNIXy to build PHARs. |
... for PHPUnit ;-) Generated some PHARs via GH Actions now with PHPUnit at These are the commit hashes I used (from oldest to most recent):
Only the last two commits I tried yield a working PHAR and both of those have the slow down. |
|
Looked at it with fresh eyes and found another cache file. Deleting that one does fix things and gets rid of the slow-down. Also, once it's gone, re-running on PHPUnit 11 (which creates the cache anew) and then PHPUnit 12 without clearing the cache in between, re-introduces the issue. Hope this gets us another step closer. |
(but probably Composer to)Summary
I'm seeing a significant performance slow-down when using PHPUnit 12 for path/branch coverage versus PHPUnit 11 when running on Windows (10).
When running just one test class (filtered down), I see a difference of 3 vs 50 seconds.
The slow-down appears to be at the start, before progress is being shown.
Current behavior
How to reproduce
If I run with
--debugon, it looks like the test suite "hangs" between "Test Passed" and "Test Finished"Reproducible via the https://github.com/Yoast/PHPUnit-Polyfills repo (though I'm still making some updates for PHPUnit 12 compatibility, which will be pulled soonish).
Expected behavior
No significant slow-down in performance.
The text was updated successfully, but these errors were encountered: