Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flaky / failing / disabled tests #8

Open
osrf-migration opened this issue Jan 2, 2019 · 12 comments
Open

Flaky / failing / disabled tests #8

osrf-migration opened this issue Jan 2, 2019 · 12 comments
Labels
bug Something isn't working macOS macOS support tests Broken or missing tests / testing infra

Comments

@osrf-migration
Copy link

osrf-migration commented Jan 2, 2019 •
edited by jacobperron
Loading

Original report (archived issue) by Louise Poubel (Bitbucket: chapulina, GitHub: chapulina).

This issue collects all known flaky tests, we should update it as we observe new flaky tests or if they stop flaking.

❄️ ❄️ ❄️

PERFORMANCE_each

Platforms: Jenkins Ubuntu and Pipelines

Failures:

39: [ RUN      ] EntityComponentManagerPerfrormance.Each
39: /opt/atlassian/pipelines/agent/build/test/performance/each.cc:141: Failure
39: Expected: (cacheEntityAvg) < (cachelessEntityAvg), actual: 7200.83 vs 7170.62
39: [  FAILED  ] EntityComponentManagerPerfrormance.Each (60561 ms)

UNIT_SimulationRunner_TEST

Platforms: Pipelines

Failures:

19: /opt/atlassian/pipelines/agent/build/src/SimulationRunner_TEST.cc:958: Failure
19: Value of: simTime.second
19:   Actual: 100000000
19: Expected: clockMsgs.back().mutable_sim()->nsec()
19: Which is: 99000000

INTEGRATION_network_handshake

Platforms: Homebrew

Failures:

/Users/jenkins/workspace/ignition_gazebo-ci-pr_any-homebrew-amd64/ign-gazebo/test/integration/network_handshake.cc:56
Value of: success
  Actual: 4-byte object <01-00 00-00>
Expected: std::cv_status::no_timeout
Which is: 4-byte object <00-00 00-00>

Platforms: Pipelines

Failures:

59: [ RUN      ] NetworkHandshake.Handshake
59: �[1;31m[Err] �[0m�[1;31m[�[0m�[1;31mLevelManager.cc�[0m�[1;31m:�[0m�[1;31m108�[0m�[1;31m] �[0m�[1;31mCould not find a plugin tag with name �[0m�[1;31mignition::gazebo�[0m�[1;31m. Levels and distributed simulation will not work.�[0m
59: �[1;31m[Err] �[0m�[1;31m[�[0m�[1;31mLevelManager.cc�[0m�[1;31m:�[0m�[1;31m108�[0m�[1;31m] �[0m�[1;31mCould not find a plugin tag with name �[0m�[1;31mignition::gazebo�[0m�[1;31m. Levels and distributed simulation will not work.�[0m
59: �[1;31m[Err] �[0m�[1;31m[�[0m�[1;31mLevelManager.cc�[0m�[1;31m:�[0m�[1;31m108�[0m�[1;31m] �[0m�[1;31mCould not find a plugin tag with name �[0m�[1;31mignition::gazebo�[0m�[1;31m. Levels and distributed simulation will not work.�[0m
59/80 Test #59: INTEGRATION_network_handshake ................***Timeout 240.01 sec

Platforms: Jenkins Ubuntu

Date added: 2019-04-11

Failures:

61: [----------] 2 tests from NetworkHandshake
61: [ RUN      ] NetworkHandshake.Handshake
61: [Msg] Loading SDF string.
61: [Dbg] [EntityComponentManager.cc:544] Using components of type [2251689575469537287] / [ign_gazebo_components.World].
61: [Dbg] [EntityComponentManager.cc:544] Using components of type [13994732549916512682] / [ign_gazebo_components.Name].
61: [Dbg] [EntityComponentManager.cc:544] Using components of type [12592746352568925681] / [ign_gazebo_components.Gravity].
61: [Dbg] [EntityComponentManager.cc:544] Using components of type [13224937992534617849] / [ign_gazebo_components.MagneticField].
61: [Dbg] [EntityComponentManager.cc:544] Using components of type [8064491505919932473] / [ign_gazebo_components.Level].
61: [Dbg] [EntityComponentManager.cc:544] Using components of type [2668898242563798256] / [ign_gazebo_components.DefaultLevel].
61: [Dbg] [EntityComponentManager.cc:544] Using components of type [11371360182141354106] / [ign_gazebo_components.LevelEntityNames].
61: [Dbg] [EntityComponentManager.cc:544] Using components of type [3297509811873971798] / [ign_gazebo_components.ParentEntity].
61: [Msg] Serving scene information on [/world/default/scene/info]
61: [Msg] Serving graph information on [/world/default/scene/graph]
61: [Msg] Serving scene information on [/world/default/scene/info]
61: [Msg] Publishing entity deletions on [/world/default/scene/deletion]
61: [Msg] Publishing pose messages on [/world/default/pose/info]
61: [Dbg] [SimulationRunner.cc:543] Loaded system [ignition::gazebo::systems::SceneBroadcaster] for entity [1]
61: [Msg] Create service on [/world/default/create]
61: [Msg] Remove service on [/world/default/remove]
61: [Dbg] [SimulationRunner.cc:543] Loaded system [ignition::gazebo::systems::UserCommands] for entity [1]
61: [Msg] Network Primary, expects [2] secondaries.
61: [Msg] Serving GUI information on [/world/default/gui/info]
61: [Msg] World [default] initialized with [default_physics] physics profile.
61: [Msg] Loading SDF string.
61: [Dbg] [EntityComponentManager.cc:544] Using components of type [2251689575469537287] / [ign_gazebo_components.World].
61: [Dbg] [EntityComponentManager.cc:544] Using components of type [13994732549916512682] / [ign_gazebo_components.Name].
61: [Dbg] [EntityComponentManager.cc:544] Using components of type [12592746352568925681] / [ign_gazebo_components.Gravity].
61: [Dbg] [EntityComponentManager.cc:544] Using components of type [13224937992534617849] / [ign_gazebo_components.MagneticField].
61: [Dbg] [EntityComponentManager.cc:544] Using components of type [8064491505919932473] / [ign_gazebo_components.Level].
61: [Dbg] [EntityComponentManager.cc:544] Using components of type [2668898242563798256] / [ign_gazebo_components.DefaultLevel].
61: [Dbg] [EntityComponentManager.cc:544] Using components of type [11371360182141354106] / [ign_gazebo_components.LevelEntityNames].
61: [Dbg] [EntityComponentManager.cc:544] Using components of type [3297509811873971798] / [ign_gazebo_components.ParentEntity].
61: [Msg] Serving scene information on [/world/default/scene/info]
61: [Msg] Serving graph information on [/world/default/scene/graph]
61: [Msg] Serving scene information on [/world/default/scene/info]
61: [Msg] Publishing entity deletions on [/world/default/scene/deletion]
61: [Msg] Publishing pose messages on [/world/default/pose/info]
61: [Dbg] [SimulationRunner.cc:543] Loaded system [ignition::gazebo::systems::SceneBroadcaster] for entity [1]
61: [Msg] Create service on [/world/default/create]
61: [Msg] Remove service on [/world/default/remove]
61: [Dbg] [SimulationRunner.cc:543] Loaded system [ignition::gazebo::systems::UserCommands] for entity [1]
61: [Dbg] [NetworkManagerSecondary.cc:49] Advertised PeerControl service on [d235de65/control]
61: [Msg] Network Secondary, with namespace [d235de65].
61: [Msg] Serving GUI information on [d235de65/world/default/gui/info]
61: [Msg] World [default] initialized with [default_physics] physics profile.
61: [Msg] Loading SDF string.
61: [Dbg] [EntityComponentManager.cc:544] Using components of type [2251689575469537287] / [ign_gazebo_components.World].
61: [Dbg] [EntityComponentManager.cc:544] Using components of type [13994732549916512682] / [ign_gazebo_components.Name].
61: [Dbg] [EntityComponentManager.cc:544] Using components of type [12592746352568925681] / [ign_gazebo_components.Gravity].
61: [Dbg] [EntityComponentManager.cc:544] Using components of type [13224937992534617849] / [ign_gazebo_components.MagneticField].
61: [Dbg] [EntityComponentManager.cc:544] Using components of type [8064491505919932473] / [ign_gazebo_components.Level].
61: [Dbg] [EntityComponentManager.cc:544] Using components of type [2668898242563798256] / [ign_gazebo_components.DefaultLevel].
61: [Dbg] [EntityComponentManager.cc:544] Using components of type [11371360182141354106] / [ign_gazebo_components.LevelEntityNames].
61: [Dbg] [EntityComponentManager.cc:544] Using components of type [3297509811873971798] / [ign_gazebo_components.ParentEntity].
61: [Msg] Serving scene information on [/world/default/scene/info�[�[1;33m[WWrn] [Componet[WWrn] [ComponetTrying to serialize copmponent of type [Trying to serialize copmponent of type [22516895754695372872251689575469537287], which hasn't implemented the `Serialize` function. Component will not be serialized.
61: ], which hasn't implemented the `Serialize` function. Component will not be serialized.
61/82 Test #61: INTEGRATION_network_handshake ................***Exception: SegFault  3.07 sec

INTEGRATION_physics_system

Platforms: Homebrew

Failures

Unable to find test results for INTEGRATION_physics_system.xml, test did not run. Expected results in /Users/jenkins/workspace/ignition_gazebo-ci-ign-gazebo3-homebrew-amd64/build/test_results/INTEGRATION_physics_system.xml

Date added: 2020-07-10

UNIT_PeerTracker_TEST

Platforms: Homebrew

Failures:

/Users/jenkins/workspace/ignition_gazebo-ci-pr_any-homebrew-amd64/ign-gazebo/src/network/PeerTracker_TEST.cc:258
Value of: tracker3.NumPeers()
  Actual: 0
Expected: 1u
Which is: 1

/Users/jenkins/workspace/ignition_gazebo-ci-pr_any-homebrew-amd64/ign-gazebo/src/network/PeerTracker_TEST.cc:84
Value of: tracker5->NumPeers()
  Actual: 2
Expected: 5u
Which is: 5

/Users/jenkins/workspace/ignition_gazebo-ci-pr_any-homebrew-amd64/ign-gazebo/src/network/PeerTracker_TEST.cc:163
Value of: stalePeers
  Actual: 0
Expected: 1

/Users/jenkins/workspace/ignition_gazebo-ci-ign-gazebo2-homebrew-amd64/ign-gazebo/src/network/PeerTracker_TEST.cc:167
Value of: tracker1->NumPeers()
  Actual: 1
Expected: 0u
Which is: 0

UNIT_NetworkManager_TEST

Platforms: Pipelines

Failures:

31: [ RUN      ] NetworkManager.Step
31: �[1;33m[Wrn] �[0m�[1;33m[�[0m�[1;33mNetworkManagerPrimary.cc�[0m�[1;33m:�[0m�[1;33m69�[0m�[1;33m] �[0m�[1;33mNetworkManager started without EventManager. �[0m�[1;33mDistributed environment may not terminate correctly�[0m�[1;33m�[0m
31/82 Test #31: UNIT_NetworkManager_TEST .....................***Exception: SegFault  1.15 sec
@osrf-migration
Copy link
Author

Original comment by Louise Poubel (Bitbucket: chapulina, GitHub: chapulina).

  • Edited issue description* changed title from "PERFORMANCE_each is flaky" to "Flaky tests"

@osrf-migration
Copy link
Author

Original comment by Louise Poubel (Bitbucket: chapulina, GitHub: chapulina).

  • Edited issue description

4 similar comments
@osrf-migration
Copy link
Author

Original comment by Louise Poubel (Bitbucket: chapulina, GitHub: chapulina).

  • Edited issue description

@osrf-migration
Copy link
Author

Original comment by Louise Poubel (Bitbucket: chapulina, GitHub: chapulina).

  • Edited issue description

@osrf-migration
Copy link
Author

Original comment by Louise Poubel (Bitbucket: chapulina, GitHub: chapulina).

  • Edited issue description

@osrf-migration
Copy link
Author

Original comment by Louise Poubel (Bitbucket: chapulina, GitHub: chapulina).

  • Edited issue description

@osrf-migration
Copy link
Author

Original comment by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).

INTEGRATION_gpu_lidar is not flaky on homebrew; it is failing consistently. I'll move it to its own issue

@osrf-migration
Copy link
Author

Original comment by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).

sometimes INTEGRATION_gpu_lidar doesn't get run because the build node doesn't have an X server running, which makes it seem like a flaky test, but it's really not

@osrf-migration
Copy link
Author

Original comment by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).

@osrf-migration osrf-migration added major bug Something isn't working labels Apr 15, 2020
@chapulina chapulina removed the major label Apr 29, 2020
@chapulina chapulina added the tests Broken or missing tests / testing infra label May 25, 2020
@azeey azeey mentioned this issue Jul 10, 2020
Merged
@jacobperron jacobperron mentioned this issue Jul 10, 2020
Merged
@chapulina chapulina changed the title Flaky tests Flaky / failing / disabled tests Aug 18, 2020
@chapulina
Copy link
Contributor

UNIT_Gui_TEST

Platforms: Homebrew

Failure: segfault at the very start

39: [----------] 1 test from GuiTest
39: [ RUN      ] GuiTest.PathManager
 39/128 Test  #39: UNIT_Gui_TEST ........................................***Exception: SegFault  0.05 sec
test 40
        Start  40: check_UNIT_Gui_TEST

@azeey
Copy link
Contributor

azeey commented Nov 17, 2020 •
edited
Loading

INTEGRATION_pose_publisher_system

Platforms: ign-gazebo3 Homebrew

Failure: Segfault on PosePublisherTest.StaticPosePublisher.

103: [ RUN      ] PosePublisherTest.StaticPosePublisher
103/134 Test #103: INTEGRATION_pose_publisher_system ....................***Exception: SegFault  5.94 sec
test 104
        Start 104: check_INTEGRATION_pose_publisher_system

INTEGRATION_multicopter

Platforms: ign-gazebo2 Homebrew

Failure: Segfault on MulticopterTest.MulticopterVelocityControl

89: [       OK ] MulticopterTest.CommandedMotorSpeed (1090 ms)
89: [ RUN      ] MulticopterTest.MulticopterVelocityControl
 89/130 Test  #89: INTEGRATION_multicopter ..............................***Exception: SegFault  2.00 sec
test 90
        Start  90: check_INTEGRATION_multicopter

@chapulina chapulina added the macOS macOS support label Nov 17, 2020
@tfoote
Copy link

tfoote commented Jul 30, 2021

ign-gazebo5 Homebrew https://build.osrfoundation.org/job/ignition_gazebo-ci-ign-gazebo5-homebrew-amd64/29/consoleFull

The following tests FAILED:
	103 - INTEGRATION_lift_drag_system (SEGFAULT)
	153 - INTEGRATION_user_commands (SEGFAULT)
Errors while running CTest

Also further documented at https://github.com/osrf/buildfarmer/issues/168 closing that in favor of this

INTEGRATION_wheel_slip
INTEGRATION_network_handshake
INTEGRATION_thermal_system

@ehfd ehfd mentioned this issue Oct 2, 2022
Closed
19 tasks
@bhagathkumar bhagathkumar mentioned this issue May 9, 2023
Open
19 tasks
@bailaC bailaC mentioned this issue May 15, 2023
Closed
6 tasks
@trexgerratt trexgerratt mentioned this issue Jun 11, 2024
Open
@DisCoLabIITK DisCoLabIITK mentioned this issue Jan 15, 2025
Open
18 tasks
@j3anm1rc j3anm1rc mentioned this issue Feb 12, 2025
Open
17 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working macOS macOS support tests Broken or missing tests / testing infra
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@azeey @tfoote @chapulina @osrf-migration