Skip to content

All registered runner tags removed after 17.9 upgrade

Summary

Support has encountered multiple customers who have had their registered runner tags disappear after upgrading to GitLab 17.9. Investigation found that the ci_runner_taggings table gets truncated.

As a workaround, the tags can be recreated by re-registering the runners, or the ci_runner_taggings table recovered from a database backup.

Steps to reproduce

Many users encountered this issue on 17.9 upgrade, but many more have not. The trigger criteria are unknown.

What is the current bug behavior?

All registered runner tags are missing after the 17.9 upgrade.

What is the expected correct behavior?

Registered runner tags are still present after the 17.9 upgrade.

Relevant logs and/or screenshots

Customer reports:

Workaround

17.9

Upgrade to http://about.gitlab.com/releases/2025/04/02/gitlab-17-9-5-released/ since it already contains the fix described below.

Open the rails console and execute:

Ci::Runner.tap { |model| model.table_name = :ci_runners_e59bb2812d }.each_batch(of: 500, column: :id) do |batch|
  scope = batch
    .joins("inner join taggings on #{Ci::Runner.quoted_table_name}.id = taggings.taggable_id")
    .where(taggings: { taggable_type: 'Ci::Runner' })
    .select(:tag_id, 'taggable_id as runner_id', :sharding_key_id, :runner_type)

  Ci::Runner.connection.execute(<<~SQL.squish)
    INSERT INTO ci_runner_taggings(tag_id, runner_id, sharding_key_id, runner_type)
    (#{scope.to_sql})
    ON CONFLICT DO NOTHING;
  SQL
end

Other solutions like #524402 (comment 2391630520) might not work because the migration numbers could be reversed (#524402 (comment 2409766761))

17.10

Upgrade to http://about.gitlab.com/releases/2025/04/02/gitlab-17-10-3-released/ since it already contains the fix described below.

Open the rails console and execute:

Ci::Runner.each_batch(of: 500) do |batch|
  scope = batch
    .joins("inner join taggings on #{Ci::Runner.quoted_table_name}.id = taggings.taggable_id")
    .where(taggings: { taggable_type: 'Ci::Runner' })
    .select(:tag_id, 'taggable_id as runner_id', :sharding_key_id, :runner_type)

  Ci::Runner.connection.execute(<<~SQL.squish)
    INSERT INTO ci_runner_taggings(tag_id, runner_id, sharding_key_id, runner_type)
    (#{scope.to_sql})
    ON CONFLICT DO NOTHING;
  SQL
end
Edited by Marius Bobin