Skip to content

Commit

Permalink
Track retries of failed database queries
Browse files Browse the repository at this point in the history
We want to have an idea on how often retries of queries are triggered.
Then we could take a decision on removing retries or not.
  • Loading branch information
eduardoj committed Oct 15, 2024
1 parent 3e8c216 commit 6b5b8b6
Show file tree
Hide file tree
Showing 3 changed files with 14 additions and 11 deletions.
7 changes: 5 additions & 2 deletions src/api/app/models/package.rb
Original file line number Diff line number Diff line change
Expand Up @@ -464,10 +464,13 @@ def sources_changed(opts = {})
begin
# NOTE: Its important that this job run in queue 'default' in order to avoid concurrency
PackageUpdateIfDirtyJob.perform_later(id)
rescue ActiveRecord::StatementInvalid
rescue ActiveRecord::StatementInvalid => e

Check warning on line 467 in src/api/app/models/package.rb

View check run for this annotation

Codecov / codecov/patch

src/api/app/models/package.rb#L467

Added line #L467 was not covered by tests
# mysql lock errors in delayed job handling... we need to retry
retries -= 1
retry if retries.positive?
if retries.positive?
Airbrake.notify("Failed while running PackageUpdateIfDirtyJob: retries left: #{retries}, package_id: #{id}, #{e}")
retry

Check warning on line 472 in src/api/app/models/package.rb

View check run for this annotation

Codecov / codecov/patch

src/api/app/models/package.rb#L470-L472

Added lines #L470 - L472 were not covered by tests
end
end
end
end
Expand Down
7 changes: 5 additions & 2 deletions src/api/app/models/package_issue.rb
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,12 @@ def self.sync_relations(package, issues)
# rubocop:enable Rails/SkipsModelValidations
end
end
rescue ActiveRecord::StatementInvalid, Mysql2::Error
rescue ActiveRecord::StatementInvalid, Mysql2::Error => e

Check warning on line 30 in src/api/app/models/package_issue.rb

View check run for this annotation

Codecov / codecov/patch

src/api/app/models/package_issue.rb#L30

Added line #L30 was not covered by tests
retries -= 1
retry if retries.positive?
if retries.positive?
Airbrake.notify("Failed to update PackageIssue : retries left: #{retries}, package: #{package.inspect}, issues: #{issues.inspect}, #{e}")
retry

Check warning on line 34 in src/api/app/models/package_issue.rb

View check run for this annotation

Codecov / codecov/patch

src/api/app/models/package_issue.rb#L32-L34

Added lines #L32 - L34 were not covered by tests
end
end
end

Expand Down
11 changes: 4 additions & 7 deletions src/api/app/models/update_notification_events.rb
Original file line number Diff line number Diff line change
Expand Up @@ -20,15 +20,12 @@ def create_events
event.save!
rescue ActiveRecord::StatementInvalid => e
retries -= 1
retry if retries.positive?
if retries.positive?
Airbrake.notify("Failed to create Event : #{type.inspect}: #{data} #{e}")
retry

Check warning on line 25 in src/api/app/models/update_notification_events.rb

View check run for this annotation

Codecov / codecov/patch

src/api/app/models/update_notification_events.rb#L23-L25

Added lines #L23 - L25 were not covered by tests
end
Airbrake.notify("Failed to create Event : #{type.inspect}: #{data} #{e}")
rescue StandardError => e
if Rails.env.test?
# make debug output useful in test suite, not just showing backtrace to Airbrake
Rails.logger.error "ERROR: #{e.inspect}: #{e.backtrace}"
Rails.logger.info e.inspect
Rails.logger.info e.backtrace
end
Airbrake.notify("Failed to create Event : #{type.inspect}: #{data} #{e}")
end
end
Expand Down

0 comments on commit 6b5b8b6

Please sign in to comment.