Server Admin Log

2024-04-18

• 18:32 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2214', diff saved to https://phabricator.wikimedia.org/P60966 and previous config saved to /var/cache/conftool/dbconfig/20240418-183211-marostegui.json
• 18:31 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1228', diff saved to https://phabricator.wikimedia.org/P60965 and previous config saved to /var/cache/conftool/dbconfig/20240418-183116-ladsgroup.json
• 18:27 dancy@deploy1002: rebuilt and synchronized wikiversions files: group2 wikis to 1.43.0-wmf.1 refs T361395
• 18:17 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2214 (T361627)', diff saved to https://phabricator.wikimedia.org/P60964 and previous config saved to /var/cache/conftool/dbconfig/20240418-181704-marostegui.json
• 18:16 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1228', diff saved to https://phabricator.wikimedia.org/P60963 and previous config saved to /var/cache/conftool/dbconfig/20240418-181606-ladsgroup.json
• 18:14 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2214 (T361627)', diff saved to https://phabricator.wikimedia.org/P60962 and previous config saved to /var/cache/conftool/dbconfig/20240418-181450-marostegui.json
• 18:14 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2214.codfw.wmnet with reason: Maintenance
• 18:14 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db2214.codfw.wmnet with reason: Maintenance
• 18:11 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2197.codfw.wmnet with reason: Maintenance
• 18:10 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db2197.codfw.wmnet with reason: Maintenance
• 18:10 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2193 (T361627)', diff saved to https://phabricator.wikimedia.org/P60961 and previous config saved to /var/cache/conftool/dbconfig/20240418-181048-marostegui.json
• 18:09 joal@deploy1002: Finished deploy [airflow-dags/analytics@980dc72]: Deploy of Analytics airflow dags for canary-events job [airflow-dags/analytics@980dc725] (duration: 00m 31s)
• 18:09 joal@deploy1002: Started deploy [airflow-dags/analytics@980dc72]: Deploy of Analytics airflow dags for canary-events job [airflow-dags/analytics@980dc725]
• 18:01 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1228 (T352010)', diff saved to https://phabricator.wikimedia.org/P60960 and previous config saved to /var/cache/conftool/dbconfig/20240418-180059-ladsgroup.json
• 17:55 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P60959 and previous config saved to /var/cache/conftool/dbconfig/20240418-175541-marostegui.json
• 17:41 joal@deploy1002: Finished deploy [airflow-dags/analytics@0a13b42]: Deploy of Analytics airflow dags for canary-events job [airflow-dags/analytics@0a13b420] (duration: 00m 28s)
• 17:41 joal@deploy1002: Started deploy [airflow-dags/analytics@0a13b42]: Deploy of Analytics airflow dags for canary-events job [airflow-dags/analytics@0a13b420]
• 17:40 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P60958 and previous config saved to /var/cache/conftool/dbconfig/20240418-174033-marostegui.json
• 17:25 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2193 (T361627)', diff saved to https://phabricator.wikimedia.org/P60957 and previous config saved to /var/cache/conftool/dbconfig/20240418-172525-marostegui.json
• 17:24 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2193 (T361627)', diff saved to https://phabricator.wikimedia.org/P60956 and previous config saved to /var/cache/conftool/dbconfig/20240418-172412-marostegui.json
• 17:24 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2193.codfw.wmnet with reason: Maintenance
• 17:23 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db2193.codfw.wmnet with reason: Maintenance
• 17:23 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T361627)', diff saved to https://phabricator.wikimedia.org/P60955 and previous config saved to /var/cache/conftool/dbconfig/20240418-172349-marostegui.json
• 17:08 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P60954 and previous config saved to /var/cache/conftool/dbconfig/20240418-170842-marostegui.json
• 16:57 kevinbazira@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
• 16:53 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P60952 and previous config saved to /var/cache/conftool/dbconfig/20240418-165334-marostegui.json
• 16:45 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on matomo1002.eqiad.wmnet with reason: Migrating to new version
• 16:44 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on matomo1002.eqiad.wmnet with reason: Migrating to new version
• 16:38 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T361627)', diff saved to https://phabricator.wikimedia.org/P60951 and previous config saved to /var/cache/conftool/dbconfig/20240418-163827-marostegui.json
• 16:36 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2180 (T361627)', diff saved to https://phabricator.wikimedia.org/P60950 and previous config saved to /var/cache/conftool/dbconfig/20240418-163612-marostegui.json
• 16:36 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2180.codfw.wmnet with reason: Maintenance
• 16:36 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db2180.codfw.wmnet with reason: Maintenance
• 16:36 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2169 (T361627)', diff saved to https://phabricator.wikimedia.org/P60949 and previous config saved to /var/cache/conftool/dbconfig/20240418-163600-marostegui.json
• 16:20 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2169', diff saved to https://phabricator.wikimedia.org/P60948 and previous config saved to /var/cache/conftool/dbconfig/20240418-162053-marostegui.json
• 16:05 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2169', diff saved to https://phabricator.wikimedia.org/P60947 and previous config saved to /var/cache/conftool/dbconfig/20240418-160546-marostegui.json
• 16:03 jforrester@deploy1002: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
• 16:03 vgutierrez: repool ncredir2001
• 16:02 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
• 16:01 jforrester@deploy1002: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
• 16:01 jforrester@deploy1002: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
• 16:01 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/admin 'apply'.
• 16:01 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
• 16:01 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
• 15:59 jforrester@deploy1002: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
• 15:59 jforrester@deploy1002: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
• 15:58 jforrester@deploy1002: helmfile [staging] START helmfile.d/services/wikifunctions: apply
• 15:57 elukey@cumin1002: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-logging-eqiad
• 15:54 jforrester@deploy1002: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
• 15:53 jforrester@deploy1002: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
• 15:53 jforrester@deploy1002: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
• 15:50 jforrester@deploy1002: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
• 15:50 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2169 (T361627)', diff saved to https://phabricator.wikimedia.org/P60946 and previous config saved to /var/cache/conftool/dbconfig/20240418-155038-marostegui.json
• 15:50 jforrester@deploy1002: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
• 15:49 jforrester@deploy1002: helmfile [staging] START helmfile.d/services/wikifunctions: apply
• 15:45 jforrester@deploy1002: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
• 15:45 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2169 (T361627)', diff saved to https://phabricator.wikimedia.org/P60945 and previous config saved to /var/cache/conftool/dbconfig/20240418-154547-marostegui.json
• 15:45 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2169.codfw.wmnet with reason: Maintenance
• 15:45 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db2169.codfw.wmnet with reason: Maintenance
• 15:45 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T361627)', diff saved to https://phabricator.wikimedia.org/P60944 and previous config saved to /var/cache/conftool/dbconfig/20240418-154524-marostegui.json
• 15:44 jforrester@deploy1002: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
• 15:44 jforrester@deploy1002: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
• 15:43 jforrester@deploy1002: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
• 15:43 jforrester@deploy1002: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
• 15:42 jforrester@deploy1002: helmfile [staging] START helmfile.d/services/wikifunctions: apply
• 15:32 moritzm: installing util-linux security updates on buster
• 15:31 elukey@cumin1002: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-logging-eqiad
• 15:30 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P60943 and previous config saved to /var/cache/conftool/dbconfig/20240418-153017-marostegui.json
• 15:26 volans: rolling python3-wmflib upgrade to 1.2.5 across the fleet
• 15:19 mforns@deploy1002: Finished deploy [airflow-dags/analytics@5fb4f99]: (no justification provided) (duration: 00m 32s)
• 15:18 mforns@deploy1002: Started deploy [airflow-dags/analytics@5fb4f99]: (no justification provided)
• 15:15 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P60942 and previous config saved to /var/cache/conftool/dbconfig/20240418-151510-marostegui.json
• 15:13 cgoubert@cumin1002: conftool action : set/weight=10:pooled=yes; selector: name=(mw1355.eqiad.wmnet|mw1480.eqiad.wmnet|mw1481.eqiad.wmnet|mw1487.eqiad.wmnet),cluster=kubernetes,service=kubesvc
• 15:12 claime: Pooling and uncordoning mw1355.eqiad.wmnet,mw1480.eqiad.wmnet,mw1481.eqiad.wmnet,mw1487.eqiad.wmnet - T351074
• 15:09 elukey@cumin2002: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching aqs2012.codfw.wmnet*: Deploy new TLS Keystore - PKI - elukey@cumin2002
• 15:04 claime: Running homer 'cr*eqiad*' commit 'T351074'
• 15:03 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1480.eqiad.wmnet with OS bullseye
• 15:02 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1355.eqiad.wmnet with OS bullseye
• 15:02 elukey@cumin2002: START - Cookbook sre.cassandra.roll-restart for nodes matching aqs2012.codfw.wmnet*: Deploy new TLS Keystore - PKI - elukey@cumin2002
• 15:00 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T361627)', diff saved to https://phabricator.wikimedia.org/P60941 and previous config saved to /var/cache/conftool/dbconfig/20240418-150003-marostegui.json
• 14:58 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1487.eqiad.wmnet with OS bullseye
• 14:56 elukey@cumin2002: END (FAIL) - Cookbook sre.cassandra.roll-restart (exit_code=99) for nodes matching aqs20[09-12]*: Deploy new TLS Keystore - PKI - elukey@cumin2002
• 14:55 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1481.eqiad.wmnet with OS bullseye
• 14:55 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2158 (T361627)', diff saved to https://phabricator.wikimedia.org/P60940 and previous config saved to /var/cache/conftool/dbconfig/20240418-145512-marostegui.json
• 14:55 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2187.codfw.wmnet with reason: Maintenance
• 14:54 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db2187.codfw.wmnet with reason: Maintenance
• 14:54 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2158.codfw.wmnet with reason: Maintenance
• 14:54 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db2158.codfw.wmnet with reason: Maintenance
• 14:54 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2151 (T361627)', diff saved to https://phabricator.wikimedia.org/P60939 and previous config saved to /var/cache/conftool/dbconfig/20240418-145435-marostegui.json
• 14:49 volans: uploaded python3-wmflib_1.2.5 to apt.wikimedia.org buster-wikimedia,bullseye-wikimedia,bookworm-wikimedia
• 14:48 moritzm: installing PHP 7.4 security updates (as packaged in Debian, not the WMF-internal build)
• 14:45 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1480.eqiad.wmnet with reason: host reimage
• 14:42 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1355.eqiad.wmnet with reason: host reimage
• 14:40 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1487.eqiad.wmnet with reason: host reimage
• 14:39 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2151', diff saved to https://phabricator.wikimedia.org/P60938 and previous config saved to /var/cache/conftool/dbconfig/20240418-143928-marostegui.json
• 14:37 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1481.eqiad.wmnet with reason: host reimage
• 14:36 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw1487.eqiad.wmnet with reason: host reimage
• 14:35 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw1355.eqiad.wmnet with reason: host reimage
• 14:35 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw1480.eqiad.wmnet with reason: host reimage
• 14:34 elukey@cumin2002: START - Cookbook sre.cassandra.roll-restart for nodes matching aqs20[09-12]*: Deploy new TLS Keystore - PKI - elukey@cumin2002
• 14:34 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw1481.eqiad.wmnet with reason: host reimage
• 14:28 elukey@cumin1002: END (ERROR) - Cookbook sre.cassandra.roll-restart (exit_code=97) for nodes matching aqs20[9-12]*: Deploy new TLS Keystore - PKI - elukey@cumin1002
• 14:28 moritzm: installing cryptsetup bugfix updates from bookworm point release
• 14:24 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2151', diff saved to https://phabricator.wikimedia.org/P60937 and previous config saved to /var/cache/conftool/dbconfig/20240418-142420-marostegui.json
• 14:22 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw1487.eqiad.wmnet with OS bullseye
• 14:22 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw1481.eqiad.wmnet with OS bullseye
• 14:21 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw1480.eqiad.wmnet with OS bullseye
• 14:21 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw1355.eqiad.wmnet with OS bullseye
• 14:19 moritzm: installing usrmerge bugfix updates from bookworm point release
• 14:12 claime: Depooling mw1355.eqiad.wmnet,mw1480.eqiad.wmnet,mw1481.eqiad.wmnet,mw1487.eqiad.wmnet - T351074
• 14:12 stevemunene@deploy1002: helmfile [eqiad] DONE helmfile.d/services/datahub: sync on main
• 14:11 Lucas_WMDE: UTC afternoon backport+config window done
• 14:11 logmsgbot: lucaswerkmeister-wmde@deploy1002 Finished scap: Backport for Added extendedconfirmed and templateeditor rights to dawiki (T361461) (duration: 16m 51s)
• 14:09 elukey@cumin1002: START - Cookbook sre.cassandra.roll-restart for nodes matching aqs20[9-12]*: Deploy new TLS Keystore - PKI - elukey@cumin1002
• 14:09 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2151 (T361627)', diff saved to https://phabricator.wikimedia.org/P60936 and previous config saved to /var/cache/conftool/dbconfig/20240418-140913-marostegui.json
• 14:08 stevemunene@deploy1002: helmfile [eqiad] START helmfile.d/services/datahub: apply on main
• 14:08 moritzm: installing postgresql-15 security updates
• 14:06 stevemunene@deploy1002: helmfile [codfw] DONE helmfile.d/services/datahub: sync on main
• 14:04 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2151 (T361627)', diff saved to https://phabricator.wikimedia.org/P60935 and previous config saved to /var/cache/conftool/dbconfig/20240418-140421-marostegui.json
• 14:04 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2151.codfw.wmnet with reason: Maintenance
• 14:04 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db2151.codfw.wmnet with reason: Maintenance
• 14:04 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2129 (T361627)', diff saved to https://phabricator.wikimedia.org/P60934 and previous config saved to /var/cache/conftool/dbconfig/20240418-140359-marostegui.json
• 14:00 stevemunene@deploy1002: helmfile [codfw] START helmfile.d/services/datahub: apply on main
• 13:59 logmsgbot: lucaswerkmeister-wmde@deploy1002 nmw03 and lucaswerkmeister-wmde: Continuing with sync
• 13:58 elukey@cumin1002: END (FAIL) - Cookbook sre.cassandra.roll-restart (exit_code=99) for nodes matching aqs20[02-12]*: Deploy new TLS Keystore - PKI - elukey@cumin1002
• 13:57 logmsgbot: lucaswerkmeister-wmde@deploy1002 nmw03 and lucaswerkmeister-wmde: Backport for Added extendedconfirmed and templateeditor rights to dawiki (T361461) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 13:56 btullis@deploy1002: helmfile [staging] DONE helmfile.d/services/datahub: sync on main
• 13:54 logmsgbot: lucaswerkmeister-wmde@deploy1002 Started scap: Backport for Added extendedconfirmed and templateeditor rights to dawiki (T361461)
• 13:52 btullis@deploy1002: helmfile [staging] START helmfile.d/services/datahub: sync on main
• 13:51 logmsgbot: lucaswerkmeister-wmde@deploy1002 Finished scap: Backport for Add 'mainpage-title-loggedin' to $wgForceUIMsgAsContentMsg (T361171) (duration: 19m 37s)
• 13:51 stevemunene@deploy1002: helmfile [codfw] DONE helmfile.d/services/datahub: sync on main
• 13:48 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2129', diff saved to https://phabricator.wikimedia.org/P60932 and previous config saved to /var/cache/conftool/dbconfig/20240418-134852-marostegui.json
• 13:48 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es1024.eqiad.wmnet
• 13:47 jynus: add grants for dbprov1005 at dbbackups (m1) T362509
• 13:40 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es1024.eqiad.wmnet
• 13:40 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es1023.eqiad.wmnet
• 13:39 moritzm: installing Linux 6.1.85 on Bookworm hosts
• 13:38 logmsgbot: lucaswerkmeister-wmde@deploy1002 lucaswerkmeister-wmde and jhsoby: Continuing with sync
• 13:37 logmsgbot: lucaswerkmeister-wmde@deploy1002 lucaswerkmeister-wmde and jhsoby: Backport for Add 'mainpage-title-loggedin' to $wgForceUIMsgAsContentMsg (T361171) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 13:37 stevemunene@deploy1002: helmfile [codfw] START helmfile.d/services/datahub: apply on main
• 13:34 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es1023.eqiad.wmnet
• 13:34 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es2025.codfw.wmnet
• 13:33 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2129', diff saved to https://phabricator.wikimedia.org/P60930 and previous config saved to /var/cache/conftool/dbconfig/20240418-133344-marostegui.json
• 13:32 logmsgbot: lucaswerkmeister-wmde@deploy1002 Started scap: Backport for Add 'mainpage-title-loggedin' to $wgForceUIMsgAsContentMsg (T361171)
• 13:28 moritzm: installing apache2 security updates
• 13:28 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es2025.codfw.wmnet
• 13:27 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es2024.codfw.wmnet
• 13:20 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2103.codfw.wmnet
• 13:20 arnaudb@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 13:20 arnaudb@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2103.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
• 13:19 arnaudb@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2103.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
• 13:18 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2129 (T361627)', diff saved to https://phabricator.wikimedia.org/P60928 and previous config saved to /var/cache/conftool/dbconfig/20240418-131836-marostegui.json
• 13:17 arnaudb@cumin1002: START - Cookbook sre.dns.netbox
• 13:14 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es2024.codfw.wmnet
• 13:13 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2129 (T361627)', diff saved to https://phabricator.wikimedia.org/P60927 and previous config saved to /var/cache/conftool/dbconfig/20240418-131311-marostegui.json
• 13:13 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2129.codfw.wmnet with reason: Maintenance
• 13:12 arnaudb@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2103.codfw.wmnet
• 13:12 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db2129.codfw.wmnet with reason: Maintenance
• 13:12 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2124 (T361627)', diff saved to https://phabricator.wikimedia.org/P60926 and previous config saved to /var/cache/conftool/dbconfig/20240418-131248-marostegui.json
• 13:10 arnaudb@cumin1002: dbctl commit (dc=all): 'db2103 depool', diff saved to https://phabricator.wikimedia.org/P60925 and previous config saved to /var/cache/conftool/dbconfig/20240418-131027-arnaudb.json
• 13:07 elukey@cumin1002: START - Cookbook sre.cassandra.roll-restart for nodes matching aqs20[02-12]*: Deploy new TLS Keystore - PKI - elukey@cumin1002
• 13:06 elukey: aqs2001's Cassandra instances moved to PKI TLS certs
• 13:01 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2105.codfw.wmnet
• 13:01 arnaudb@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 13:01 arnaudb@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2105.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
• 13:00 arnaudb@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2105.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
• 12:58 arnaudb@cumin1002: START - Cookbook sre.dns.netbox
• 12:57 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2124', diff saved to https://phabricator.wikimedia.org/P60923 and previous config saved to /var/cache/conftool/dbconfig/20240418-125739-marostegui.json
• 12:55 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es2023.codfw.wmnet
• 12:54 arnaudb@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2105.codfw.wmnet
• 12:54 sukhe: sudo cumin -b1 -s600 "A:dnsbox" "systemctl restart ntp.service" to pick up magru /24: T346722
• 12:53 arnaudb@cumin1002: dbctl commit (dc=all): 'db2105 depool', diff saved to https://phabricator.wikimedia.org/P60922 and previous config saved to /var/cache/conftool/dbconfig/20240418-125338-arnaudb.json
• 12:49 elukey: move aqs codfw cassandra instances to PKI TLS certs - T352647
• 12:45 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2106.codfw.wmnet
• 12:45 arnaudb@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:45 arnaudb@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2106.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
• 12:27 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2124 (T361627)', diff saved to https://phabricator.wikimedia.org/P60919 and previous config saved to /var/cache/conftool/dbconfig/20240418-122721-marostegui.json
• 12:26 arnaudb@cumin1002: START - Cookbook sre.dns.netbox
• 12:22 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2124 (T361627)', diff saved to https://phabricator.wikimedia.org/P60918 and previous config saved to /var/cache/conftool/dbconfig/20240418-122227-marostegui.json
• 12:22 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2124.codfw.wmnet with reason: Maintenance
• 12:22 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db2124.codfw.wmnet with reason: Maintenance
• 12:21 arnaudb@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2107.codfw.wmnet
• 12:21 arnaudb@cumin1002: dbctl commit (dc=all): 'db2107 depool', diff saved to https://phabricator.wikimedia.org/P60917 and previous config saved to /var/cache/conftool/dbconfig/20240418-122122-arnaudb.json
• 12:18 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2097.codfw.wmnet with reason: Maintenance
• 12:18 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db2097.codfw.wmnet with reason: Maintenance
• 12:16 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
• 12:16 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
• 12:16 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1231 (T361627)', diff saved to https://phabricator.wikimedia.org/P60916 and previous config saved to /var/cache/conftool/dbconfig/20240418-121559-marostegui.json
• 12:15 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host crm2001.codfw.wmnet
• 12:14 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host crm2001.codfw.wmnet
• 12:14 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host matomo1003.eqiad.wmnet
• 12:13 isaranto@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
• 12:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host matomo1003.eqiad.wmnet
• 12:08 vgutierrez: depool ncredir2001
• 12:06 eoghan: Switching phab1004 to use cfssl issued ssl cert https://gerrit.wikimedia.org/r/c/operations/puppet/+/1020190
• 12:02 moritzm: installing PHP 8.2 security updates
• 12:00 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P60915 and previous config saved to /var/cache/conftool/dbconfig/20240418-120051-marostegui.json
• 12:00 isaranto@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
• 11:56 isaranto@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
• 11:54 moritzm: upgrading PHP security updates on eqiad baremetal servers T362511
• 11:52 cgoubert@cumin1002: conftool action : set/weight=10:pooled=yes; selector: name=(mw2302.codfw.wmnet|mw2303.codfw.wmnet|mw2304.codfw.wmnet|mw2332.codfw.wmnet|mw2333.codfw.wmnet|mw2334.codfw.wmnet),cluster=kubernetes,service=kubesvc
• 11:52 claime: Pooling and uncordoning mw2302.codfw.wmnet,mw2303.codfw.wmnet,mw2304.codfw.wmnet,mw2332.codfw.wmnet,mw2333.codfw.wmnet,mw2334.codfw.wmnet - T351074
• 11:45 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P60914 and previous config saved to /var/cache/conftool/dbconfig/20240418-114544-marostegui.json
• 11:42 claime: Running homer 'cr*codfw*' commit 'T351074'
• 11:35 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2333.codfw.wmnet with OS bullseye
• 11:33 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
• 11:32 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
• 11:32 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
• 11:31 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
• 11:30 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1231 (T361627)', diff saved to https://phabricator.wikimedia.org/P60913 and previous config saved to /var/cache/conftool/dbconfig/20240418-113037-marostegui.json
• 11:30 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2302.codfw.wmnet with OS bullseye
• 11:29 cgoubert@deploy1002: Finished scap: Redeploy mw-on-k8s with full rebuild - Fix setting php.timeout - T358308 (duration: 37m 04s)
• 11:28 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1231 (T361627)', diff saved to https://phabricator.wikimedia.org/P60912 and previous config saved to /var/cache/conftool/dbconfig/20240418-112827-marostegui.json
• 11:28 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1172 (T352010)', diff saved to https://phabricator.wikimedia.org/P60911 and previous config saved to /var/cache/conftool/dbconfig/20240418-112816-ladsgroup.json
• 11:28 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1231.eqiad.wmnet with reason: Maintenance
• 11:28 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1172.eqiad.wmnet with reason: Maintenance
• 11:28 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db1231.eqiad.wmnet with reason: Maintenance
• 11:27 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1172.eqiad.wmnet with reason: Maintenance
• 11:26 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es1021.eqiad.wmnet
• 11:25 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1225.eqiad.wmnet with reason: Maintenance
• 11:25 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2304.codfw.wmnet with OS bullseye
• 11:25 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db1225.eqiad.wmnet with reason: Maintenance
• 11:25 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1224 (T361627)', diff saved to https://phabricator.wikimedia.org/P60910 and previous config saved to /var/cache/conftool/dbconfig/20240418-112459-marostegui.json
• 11:23 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2334.codfw.wmnet with OS bullseye
• 11:20 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2332.codfw.wmnet with OS bullseye
• 11:18 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es1021.eqiad.wmnet
• 11:16 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2303.codfw.wmnet with OS bullseye
• 11:13 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es1020.eqiad.wmnet
• 11:11 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1183 (T355609)', diff saved to https://phabricator.wikimedia.org/P60909 and previous config saved to /var/cache/conftool/dbconfig/20240418-111132-marostegui.json
• 11:10 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2302.codfw.wmnet with reason: host reimage
• 11:09 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1224', diff saved to https://phabricator.wikimedia.org/P60908 and previous config saved to /var/cache/conftool/dbconfig/20240418-110950-marostegui.json
• 11:08 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2333.codfw.wmnet with reason: host reimage
• 11:05 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2304.codfw.wmnet with reason: host reimage
• 11:03 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es1020.eqiad.wmnet
• 11:02 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2334.codfw.wmnet with reason: host reimage
• 11:01 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es2022.codfw.wmnet
• 11:00 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2332.codfw.wmnet with reason: host reimage
• 10:57 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2303.codfw.wmnet with reason: host reimage
• 10:56 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1183', diff saved to https://phabricator.wikimedia.org/P60907 and previous config saved to /var/cache/conftool/dbconfig/20240418-105624-marostegui.json
• 10:56 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw2334.codfw.wmnet with reason: host reimage
• 10:55 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw2333.codfw.wmnet with reason: host reimage
• 10:55 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw2304.codfw.wmnet with reason: host reimage
• 10:54 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw2332.codfw.wmnet with reason: host reimage
• 10:54 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1224', diff saved to https://phabricator.wikimedia.org/P60906 and previous config saved to /var/cache/conftool/dbconfig/20240418-105441-marostegui.json
• 10:54 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw2302.codfw.wmnet with reason: host reimage
• 10:54 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw2303.codfw.wmnet with reason: host reimage
• 10:52 cgoubert@deploy1002: Started scap: Redeploy mw-on-k8s with full rebuild - Fix setting php.timeout - T358308
• 10:52 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es2022.codfw.wmnet
• 10:48 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es2021.codfw.wmnet
• 10:45 claime: Rebuild php7.4-fpm production images - T358308
• 10:41 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1183', diff saved to https://phabricator.wikimedia.org/P60905 and previous config saved to /var/cache/conftool/dbconfig/20240418-104117-marostegui.json
• 10:40 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es2021.codfw.wmnet
• 10:39 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw2334.codfw.wmnet with OS bullseye
• 10:39 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1224 (T361627)', diff saved to https://phabricator.wikimedia.org/P60904 and previous config saved to /var/cache/conftool/dbconfig/20240418-103933-marostegui.json
• 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es2020.codfw.wmnet
• 10:39 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw2333.codfw.wmnet with OS bullseye
• 10:38 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw2332.codfw.wmnet with OS bullseye
• 10:38 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw2304.codfw.wmnet with OS bullseye
• 10:38 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw2303.codfw.wmnet with OS bullseye
• 10:37 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw2302.codfw.wmnet with OS bullseye
• 10:34 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1224 (T361627)', diff saved to https://phabricator.wikimedia.org/P60903 and previous config saved to /var/cache/conftool/dbconfig/20240418-103422-marostegui.json
• 10:34 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1224.eqiad.wmnet with reason: Maintenance
• 10:34 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db1224.eqiad.wmnet with reason: Maintenance
• 10:34 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T361627)', diff saved to https://phabricator.wikimedia.org/P60902 and previous config saved to /var/cache/conftool/dbconfig/20240418-103359-marostegui.json
• 10:30 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es2020.codfw.wmnet
• 10:26 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1183 (T355609)', diff saved to https://phabricator.wikimedia.org/P60901 and previous config saved to /var/cache/conftool/dbconfig/20240418-102609-marostegui.json
• 10:25 claime: Depooling mw2302.codfw.wmnet,mw2303.codfw.wmnet,mw2304.codfw.wmnet,mw2332.codfw.wmnet,mw2333.codfw.wmnet,mw2334.codfw.wmnet for reimage - T351074
• 10:18 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P60900 and previous config saved to /var/cache/conftool/dbconfig/20240418-101852-marostegui.json
• 10:18 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1183 (T355609)', diff saved to https://phabricator.wikimedia.org/P60899 and previous config saved to /var/cache/conftool/dbconfig/20240418-101841-marostegui.json
• 10:18 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1183.eqiad.wmnet with reason: Maintenance
• 10:18 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1183.eqiad.wmnet with reason: Maintenance
• 10:08 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host contint1002.wikimedia.org
• 10:03 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P60898 and previous config saved to /var/cache/conftool/dbconfig/20240418-100338-marostegui.json
• 09:57 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host contint1002.wikimedia.org
• 09:54 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1087.eqiad.wmnet
• 09:53 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1228 (T352010)', diff saved to https://phabricator.wikimedia.org/P60897 and previous config saved to /var/cache/conftool/dbconfig/20240418-095331-ladsgroup.json
• 09:53 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1228.eqiad.wmnet with reason: Maintenance
• 09:53 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1228.eqiad.wmnet with reason: Maintenance
• 09:53 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1219 (T352010)', diff saved to https://phabricator.wikimedia.org/P60896 and previous config saved to /var/cache/conftool/dbconfig/20240418-095308-ladsgroup.json
• 09:48 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T361627)', diff saved to https://phabricator.wikimedia.org/P60895 and previous config saved to /var/cache/conftool/dbconfig/20240418-094830-marostegui.json
• 09:46 btullis@cumin1002: START - Cookbook sre.hosts.reboot-single for host an-worker1087.eqiad.wmnet
• 09:46 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1201 (T361627)', diff saved to https://phabricator.wikimedia.org/P60894 and previous config saved to /var/cache/conftool/dbconfig/20240418-094619-marostegui.json
• 09:46 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1201.eqiad.wmnet with reason: Maintenance
• 09:46 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db1201.eqiad.wmnet with reason: Maintenance
• 09:45 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T361627)', diff saved to https://phabricator.wikimedia.org/P60893 and previous config saved to /var/cache/conftool/dbconfig/20240418-094556-marostegui.json
• 09:42 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1183 (T356166)', diff saved to https://phabricator.wikimedia.org/P60892 and previous config saved to /var/cache/conftool/dbconfig/20240418-094235-marostegui.json
• 09:39 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es1034.eqiad.wmnet
• 09:38 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P60891 and previous config saved to /var/cache/conftool/dbconfig/20240418-093759-ladsgroup.json
• 09:35 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2108.codfw.wmnet
• 09:35 arnaudb@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 09:35 arnaudb@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2108.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
• 09:34 arnaudb@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2108.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
• 09:32 arnaudb@cumin1002: START - Cookbook sre.dns.netbox
• 09:30 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P60890 and previous config saved to /var/cache/conftool/dbconfig/20240418-093049-marostegui.json
• 09:27 arnaudb@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2108.codfw.wmnet
• 09:27 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1183', diff saved to https://phabricator.wikimedia.org/P60889 and previous config saved to /var/cache/conftool/dbconfig/20240418-092728-marostegui.json
• 09:27 arnaudb@cumin1002: dbctl commit (dc=all): 'Depool db2108', diff saved to https://phabricator.wikimedia.org/P60888 and previous config saved to /var/cache/conftool/dbconfig/20240418-092718-arnaudb.json
• 09:25 mforns@deploy1002: Finished deploy [analytics/refinery@be07da9]: Regular analytics weekly train [analytics/refinery@be07da9e] (duration: 00m 20s)
• 09:25 mforns@deploy1002: Started deploy [analytics/refinery@be07da9]: Regular analytics weekly train [analytics/refinery@be07da9e]
• 09:25 arnaudb@cumin1002: dbctl commit (dc=all): 'Depool db2108', diff saved to https://phabricator.wikimedia.org/P60887 and previous config saved to /var/cache/conftool/dbconfig/20240418-092504-arnaudb.json
• 09:24 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es1034.eqiad.wmnet
• 09:22 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P60886 and previous config saved to /var/cache/conftool/dbconfig/20240418-092252-ladsgroup.json
• 09:22 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es1028.eqiad.wmnet
• 09:20 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2109.codfw.wmnet
• 09:20 arnaudb@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 09:20 arnaudb@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2109.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
• 09:19 arnaudb@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2109.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
• 09:17 arnaudb@cumin1002: START - Cookbook sre.dns.netbox
• 09:15 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P60885 and previous config saved to /var/cache/conftool/dbconfig/20240418-091541-marostegui.json
• 09:13 arnaudb@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2109.codfw.wmnet
• 09:12 arnaudb@cumin1002: dbctl commit (dc=all): 'Depool db2109', diff saved to https://phabricator.wikimedia.org/P60884 and previous config saved to /var/cache/conftool/dbconfig/20240418-091251-arnaudb.json
• 09:11 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1183', diff saved to https://phabricator.wikimedia.org/P60883 and previous config saved to /var/cache/conftool/dbconfig/20240418-091126-marostegui.json
• 09:09 mforns@deploy1002: Finished deploy [analytics/refinery@be07da9] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@be07da9e] (duration: 02m 46s)
• 09:08 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es1028.eqiad.wmnet
• 09:07 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es2034.codfw.wmnet
• 09:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1219 (T352010)', diff saved to https://phabricator.wikimedia.org/P60882 and previous config saved to /var/cache/conftool/dbconfig/20240418-090744-ladsgroup.json
• 09:06 mforns@deploy1002: Started deploy [analytics/refinery@be07da9] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@be07da9e]
• 09:06 mforns@deploy1002: Finished deploy [analytics/refinery@be07da9] (thin): Regular analytics weekly train THIN [analytics/refinery@be07da9e] (duration: 03m 45s)
• 09:04 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2110.codfw.wmnet
• 09:04 arnaudb@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 09:04 arnaudb@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2110.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
• 09:03 arnaudb@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2110.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
• 09:02 mforns@deploy1002: Started deploy [analytics/refinery@be07da9] (thin): Regular analytics weekly train THIN [analytics/refinery@be07da9e]
• 09:01 arnaudb@cumin1002: START - Cookbook sre.dns.netbox
• 09:00 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T361627)', diff saved to https://phabricator.wikimedia.org/P60881 and previous config saved to /var/cache/conftool/dbconfig/20240418-090032-marostegui.json
• 08:59 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1187 (T361627)', diff saved to https://phabricator.wikimedia.org/P60880 and previous config saved to /var/cache/conftool/dbconfig/20240418-085922-marostegui.json
• 08:59 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1187.eqiad.wmnet with reason: Maintenance
• 08:59 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db1187.eqiad.wmnet with reason: Maintenance
• 08:59 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T361627)', diff saved to https://phabricator.wikimedia.org/P60879 and previous config saved to /var/cache/conftool/dbconfig/20240418-085900-marostegui.json
• 08:57 arnaudb@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2110.codfw.wmnet
• 08:57 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es2034.codfw.wmnet
• 08:56 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1183 (T356166)', diff saved to https://phabricator.wikimedia.org/P60878 and previous config saved to /var/cache/conftool/dbconfig/20240418-085619-marostegui.json
• 08:56 arnaudb@cumin1002: dbctl commit (dc=all): 'Depool db2110', diff saved to https://phabricator.wikimedia.org/P60877 and previous config saved to /var/cache/conftool/dbconfig/20240418-085608-arnaudb.json
• 08:52 arnaudb@cumin1002: dbctl commit (dc=all): 'db2110 depool', diff saved to https://phabricator.wikimedia.org/P60876 and previous config saved to /var/cache/conftool/dbconfig/20240418-085235-arnaudb.json
• 08:45 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1183 (T356166)', diff saved to https://phabricator.wikimedia.org/P60875 and previous config saved to /var/cache/conftool/dbconfig/20240418-084510-marostegui.json
• 08:45 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1183.eqiad.wmnet with reason: Maintenance
• 08:45 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1183.eqiad.wmnet with reason: Maintenance
• 08:43 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P60874 and previous config saved to /var/cache/conftool/dbconfig/20240418-084353-marostegui.json
• 08:42 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es2029.codfw.wmnet
• 08:42 arnaudb@cumin1002: dbctl commit (dc=all): 'db1183 (re)pooling @ 100%: post maintenance repool', diff saved to https://phabricator.wikimedia.org/P60873 and previous config saved to /var/cache/conftool/dbconfig/20240418-084223-arnaudb.json
• 08:41 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2111.codfw.wmnet
• 08:41 arnaudb@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 08:41 arnaudb@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2111.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
• 08:41 mforns@deploy1002: Finished deploy [analytics/refinery@be07da9]: Regular analytics weekly train [analytics/refinery@be07da9e] (duration: 00m 15s)
• 08:41 mforns@deploy1002: Started deploy [analytics/refinery@be07da9]: Regular analytics weekly train [analytics/refinery@be07da9e]
• 08:40 arnaudb@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2111.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
• 08:38 arnaudb@cumin1002: START - Cookbook sre.dns.netbox
• 08:34 arnaudb@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2111.codfw.wmnet
• 08:34 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es2029.codfw.wmnet
• 08:34 arnaudb@cumin1002: dbctl commit (dc=all): 'db2111 depool', diff saved to https://phabricator.wikimedia.org/P60872 and previous config saved to /var/cache/conftool/dbconfig/20240418-083422-arnaudb.json
• 08:34 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es2027.codfw.wmnet
• 08:32 arnaudb@cumin1002: dbctl commit (dc=all): 'db2111 depool', diff saved to https://phabricator.wikimedia.org/P60871 and previous config saved to /var/cache/conftool/dbconfig/20240418-083245-arnaudb.json
• 08:28 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P60870 and previous config saved to /var/cache/conftool/dbconfig/20240418-082845-marostegui.json
• 08:27 arnaudb@cumin1002: dbctl commit (dc=all): 'db1183 (re)pooling @ 75%: post maintenance repool', diff saved to https://phabricator.wikimedia.org/P60869 and previous config saved to /var/cache/conftool/dbconfig/20240418-082717-arnaudb.json
• 08:27 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2112.codfw.wmnet
• 08:27 arnaudb@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 08:26 arnaudb@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2112.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
• 08:25 arnaudb@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2112.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
• 08:25 mforns@deploy1002: Finished deploy [analytics/refinery@be07da9]: Regular analytics weekly train [analytics/refinery@be07da9e] (duration: 14m 07s)
• 08:24 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es2027.codfw.wmnet
• 08:23 arnaudb@cumin1002: START - Cookbook sre.dns.netbox
• 08:15 arnaudb@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2112.codfw.wmnet
• 08:14 arnaudb@cumin1002: dbctl commit (dc=all): 'db2112 depool', diff saved to https://phabricator.wikimedia.org/P60867 and previous config saved to /var/cache/conftool/dbconfig/20240418-081439-arnaudb.json
• 08:13 kostajh: UTC morning deploys done
• 08:13 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T361627)', diff saved to https://phabricator.wikimedia.org/P60866 and previous config saved to /var/cache/conftool/dbconfig/20240418-081338-marostegui.json
• 08:12 arnaudb@cumin1002: dbctl commit (dc=all): 'db1183 (re)pooling @ 50%: post maintenance repool', diff saved to https://phabricator.wikimedia.org/P60865 and previous config saved to /var/cache/conftool/dbconfig/20240418-081210-arnaudb.json
• 08:11 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1180 (T361627)', diff saved to https://phabricator.wikimedia.org/P60864 and previous config saved to /var/cache/conftool/dbconfig/20240418-081127-marostegui.json
• 08:11 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1180.eqiad.wmnet with reason: Maintenance
• 08:11 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db1180.eqiad.wmnet with reason: Maintenance
• 08:11 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T361627)', diff saved to https://phabricator.wikimedia.org/P60863 and previous config saved to /var/cache/conftool/dbconfig/20240418-081104-marostegui.json
• 08:11 mforns@deploy1002: Started deploy [analytics/refinery@be07da9]: Regular analytics weekly train [analytics/refinery@be07da9e]
• 08:10 kharlan@deploy1002: Finished scap: Backport for EventStreamConfig: Fix stream title for mediawiki.ip_reputation.score (T354597) (duration: 19m 36s)
• 08:07 isaranto@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
• 08:02 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2113.codfw.wmnet
• 08:02 arnaudb@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 08:02 arnaudb@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2113.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
• 08:00 arnaudb@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2113.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
• 07:58 kharlan@deploy1002: urbanecm and kharlan: Continuing with sync
• 07:57 arnaudb@cumin1002: START - Cookbook sre.dns.netbox
• 07:57 arnaudb@cumin1002: dbctl commit (dc=all): 'db1183 (re)pooling @ 25%: post maintenance repool', diff saved to https://phabricator.wikimedia.org/P60862 and previous config saved to /var/cache/conftool/dbconfig/20240418-075704-arnaudb.json
• 07:55 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P60861 and previous config saved to /var/cache/conftool/dbconfig/20240418-075557-marostegui.json
• 07:54 kharlan@deploy1002: urbanecm and kharlan: Backport for EventStreamConfig: Fix stream title for mediawiki.ip_reputation.score (T354597) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 07:52 arnaudb@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2113.codfw.wmnet
• 07:51 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 depool', diff saved to https://phabricator.wikimedia.org/P60860 and previous config saved to /var/cache/conftool/dbconfig/20240418-075154-arnaudb.json
• 07:51 kharlan@deploy1002: Started scap: Backport for EventStreamConfig: Fix stream title for mediawiki.ip_reputation.score (T354597)
• 07:47 urbanecm@deploy1002: Finished scap: Backport for WikimediaEvents: Set IPoid URL and enable ip_reputation/score (2nd attempt) (T354597), ext-EventLogging: Add mediawiki.ip_reputation.score (T354597) (duration: 22m 27s)
• 07:41 arnaudb@cumin1002: dbctl commit (dc=all): 'db1183 (re)pooling @ 15%: post maintenance repool', diff saved to https://phabricator.wikimedia.org/P60859 and previous config saved to /var/cache/conftool/dbconfig/20240418-074158-arnaudb.json
• 07:40 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P60858 and previous config saved to /var/cache/conftool/dbconfig/20240418-074049-marostegui.json
• 07:34 urbanecm@deploy1002: kharlan and urbanecm: Continuing with sync
• 07:31 moritzm: upgrading PHP security updates on codfw baremetal servers T362511
• 07:28 urbanecm@deploy1002: kharlan and urbanecm: Backport for WikimediaEvents: Set IPoid URL and enable ip_reputation/score (2nd attempt) (T354597), ext-EventLogging: Add mediawiki.ip_reputation.score (T354597) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 07:26 arnaudb@cumin1002: dbctl commit (dc=all): 'db1183 (re)pooling @ 10%: post maintenance repool', diff saved to https://phabricator.wikimedia.org/P60857 and previous config saved to /var/cache/conftool/dbconfig/20240418-072653-arnaudb.json
• 07:25 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T361627)', diff saved to https://phabricator.wikimedia.org/P60856 and previous config saved to /var/cache/conftool/dbconfig/20240418-072542-marostegui.json
• 07:25 urbanecm@deploy1002: Started scap: Backport for WikimediaEvents: Set IPoid URL and enable ip_reputation/score (2nd attempt) (T354597), ext-EventLogging: Add mediawiki.ip_reputation.score (T354597)
• 07:24 marostegui@cumin1002: dbctl commit (dc=all): 'db2108 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P60855 and previous config saved to /var/cache/conftool/dbconfig/20240418-072410-root.json
• 07:23 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1168 (T361627)', diff saved to https://phabricator.wikimedia.org/P60854 and previous config saved to /var/cache/conftool/dbconfig/20240418-072331-marostegui.json
• 07:23 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1168.eqiad.wmnet with reason: Maintenance
• 07:23 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db1168.eqiad.wmnet with reason: Maintenance
• 07:23 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T361627)', diff saved to https://phabricator.wikimedia.org/P60853 and previous config saved to /var/cache/conftool/dbconfig/20240418-072309-marostegui.json
• 07:21 urbanecm@deploy1002: Finished scap: Backport for [plwiki] Limit Content Translation publishing to mainspace for non-editors (T362756) (duration: 17m 15s)
• 07:11 arnaudb@cumin1002: dbctl commit (dc=all): 'db1183 (re)pooling @ 5%: post maintenance repool', diff saved to https://phabricator.wikimedia.org/P60852 and previous config saved to /var/cache/conftool/dbconfig/20240418-071147-arnaudb.json
• 07:09 marostegui@cumin1002: dbctl commit (dc=all): 'db2108 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P60851 and previous config saved to /var/cache/conftool/dbconfig/20240418-070904-root.json
• 07:08 urbanecm@deploy1002: msz2001 and urbanecm: Continuing with sync
• 07:08 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P60850 and previous config saved to /var/cache/conftool/dbconfig/20240418-070801-marostegui.json
• 07:07 urbanecm@deploy1002: msz2001 and urbanecm: Backport for [plwiki] Limit Content Translation publishing to mainspace for non-editors (T362756) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 07:04 urbanecm@deploy1002: Started scap: Backport for [plwiki] Limit Content Translation publishing to mainspace for non-editors (T362756)
• 06:56 arnaudb@cumin1002: dbctl commit (dc=all): 'db1183 (re)pooling @ 2%: post maintenance repool', diff saved to https://phabricator.wikimedia.org/P60849 and previous config saved to /var/cache/conftool/dbconfig/20240418-065641-arnaudb.json
• 06:53 marostegui@cumin1002: dbctl commit (dc=all): 'db2108 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P60848 and previous config saved to /var/cache/conftool/dbconfig/20240418-065358-root.json
• 06:52 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P60847 and previous config saved to /var/cache/conftool/dbconfig/20240418-065254-marostegui.json
• 06:41 arnaudb@cumin1002: dbctl commit (dc=all): 'db1183 (re)pooling @ 1%: post maintenance repool', diff saved to https://phabricator.wikimedia.org/P60846 and previous config saved to /var/cache/conftool/dbconfig/20240418-064135-arnaudb.json
• 06:39 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1183.eqiad.wmnet with reason: Maintenance
• 06:39 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1183.eqiad.wmnet with reason: Maintenance
• 06:38 marostegui@cumin1002: dbctl commit (dc=all): 'db2108 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P60845 and previous config saved to /var/cache/conftool/dbconfig/20240418-063852-root.json
• 06:37 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T361627)', diff saved to https://phabricator.wikimedia.org/P60844 and previous config saved to /var/cache/conftool/dbconfig/20240418-063746-marostegui.json
• 06:36 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1183.eqiad.wmnet with OS bookworm
• 06:35 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1165 (T361627)', diff saved to https://phabricator.wikimedia.org/P60843 and previous config saved to /var/cache/conftool/dbconfig/20240418-063536-marostegui.json
• 06:35 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
• 06:35 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
• 06:35 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1165.eqiad.wmnet with reason: Maintenance
• 06:34 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db1165.eqiad.wmnet with reason: Maintenance
• 06:30 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
• 06:30 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
• 06:23 marostegui@cumin1002: dbctl commit (dc=all): 'db2108 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P60842 and previous config saved to /var/cache/conftool/dbconfig/20240418-062346-root.json
• 06:15 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1183.eqiad.wmnet with reason: host reimage
• 06:13 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1183.eqiad.wmnet with reason: host reimage
• 06:08 marostegui@cumin1002: dbctl commit (dc=all): 'db2108 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P60841 and previous config saved to /var/cache/conftool/dbconfig/20240418-060841-root.json
• 06:02 arnaudb@cumin1002: START - Cookbook sre.hosts.reimage for host db1183.eqiad.wmnet with OS bookworm
• 06:00 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2205.codfw.wmnet with reason: Maintenance
• 06:00 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db2205.codfw.wmnet with reason: Maintenance
• 05:57 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1183.eqiad.wmnet with reason: upgrade db1183 T360116
• 05:57 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1183.eqiad.wmnet with reason: upgrade db1183 T360116
• 05:56 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2108.codfw.wmnet with OS bookworm
• 05:53 marostegui@cumin1002: dbctl commit (dc=all): 'db2108 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P60840 and previous config saved to /var/cache/conftool/dbconfig/20240418-055335-root.json
• 05:50 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
• 05:50 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
• 05:42 arnaudb@cumin1002: dbctl commit (dc=all): 'Depool db1183 T362668', diff saved to https://phabricator.wikimedia.org/P60838 and previous config saved to /var/cache/conftool/dbconfig/20240418-054247-arnaudb.json
• 05:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Promote db1230 to s5 primary and set section read-write T362668', diff saved to https://phabricator.wikimedia.org/P60837 and previous config saved to /var/cache/conftool/dbconfig/20240418-053852-arnaudb.json
• 05:36 arnaudb@cumin1002: dbctl commit (dc=all): 'Set s5 eqiad as read-only for maintenance - T362668', diff saved to https://phabricator.wikimedia.org/P60836 and previous config saved to /var/cache/conftool/dbconfig/20240418-053657-arnaudb.json
• 05:35 arnaudb: Starting s5 eqiad failover from db1183 to db1230 - T362668
• 05:34 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2108.codfw.wmnet with reason: host reimage
• 05:31 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2108.codfw.wmnet with reason: host reimage
• 05:20 marostegui: dbmaint Upgrade s7 codfw to Bookworm and MariaDB 10.6 T362745
• 05:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Set db1230 with weight 0 T362668', diff saved to https://phabricator.wikimedia.org/P60835 and previous config saved to /var/cache/conftool/dbconfig/20240418-051639-arnaudb.json
• 05:16 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 26 hosts with reason: Primary switchover s5 T362668
• 05:16 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on 26 hosts with reason: Primary switchover s5 T362668
• 05:13 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db2108.codfw.wmnet with OS bookworm
• 05:11 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2108', diff saved to https://phabricator.wikimedia.org/P60834 and previous config saved to /var/cache/conftool/dbconfig/20240418-051129-root.json
• 00:06 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1219 (T352010)', diff saved to https://phabricator.wikimedia.org/P60833 and previous config saved to /var/cache/conftool/dbconfig/20240418-000639-ladsgroup.json
• 00:06 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1219.eqiad.wmnet with reason: Maintenance
• 00:06 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1219.eqiad.wmnet with reason: Maintenance
• 00:06 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1218 (T352010)', diff saved to https://phabricator.wikimedia.org/P60832 and previous config saved to /var/cache/conftool/dbconfig/20240418-000616-ladsgroup.json

2024-04-17

• 23:51 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1218', diff saved to https://phabricator.wikimedia.org/P60831 and previous config saved to /var/cache/conftool/dbconfig/20240417-235105-ladsgroup.json
• 23:48 amastilovic@deploy1002: Finished deploy [airflow-dags/analytics@c9d6969]: (no justification provided) (duration: 00m 37s)
• 23:47 amastilovic@deploy1002: Started deploy [airflow-dags/analytics@c9d6969]: (no justification provided)
• 23:37 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1171.eqiad.wmnet with reason: Maintenance
• 23:37 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1171.eqiad.wmnet with reason: Maintenance
• 23:37 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167 (T352010)', diff saved to https://phabricator.wikimedia.org/P60830 and previous config saved to /var/cache/conftool/dbconfig/20240417-233731-ladsgroup.json
• 23:35 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1218', diff saved to https://phabricator.wikimedia.org/P60829 and previous config saved to /var/cache/conftool/dbconfig/20240417-233557-ladsgroup.json
• 23:22 sukhe: sukhe@cp1114:~$ sudo -i haproxy-restart
• 23:22 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P60828 and previous config saved to /var/cache/conftool/dbconfig/20240417-232221-ladsgroup.json
• 23:20 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1218 (T352010)', diff saved to https://phabricator.wikimedia.org/P60827 and previous config saved to /var/cache/conftool/dbconfig/20240417-232050-ladsgroup.json
• 23:14 mutante: rsyncing jenkins data from contint2002 to contint1002, pre-sync in preparation for migration next week - /srv/jenkins (291G) and much smaller zuul and jenkins data dirs T334517
• 23:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P60826 and previous config saved to /var/cache/conftool/dbconfig/20240417-230714-ladsgroup.json
• 22:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167 (T352010)', diff saved to https://phabricator.wikimedia.org/P60825 and previous config saved to /var/cache/conftool/dbconfig/20240417-225206-ladsgroup.json
• 22:42 zabe@deploy1002: Finished scap: Backport for Revert "REST: Deprecate using "post" as the parameter source" (T362817) (duration: 17m 14s)
• 22:29 zabe@deploy1002: jforrester and zabe: Continuing with sync
• 22:28 zabe@deploy1002: jforrester and zabe: Backport for Revert "REST: Deprecate using "post" as the parameter source" (T362817) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 22:24 zabe@deploy1002: Started scap: Backport for Revert "REST: Deprecate using "post" as the parameter source" (T362817)
• 22:11 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 19 hosts with reason: T362508
• 22:10 bking@cumin2002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on 19 hosts with reason: T362508
• 21:50 mutante: deploying scap config change (gerrit:1020321) - [cumin2002:~] $ sudo cumin -b 4 -s 40 'C:scap AND mw*' 'run-puppet-agent' T359643
• 21:09 mutante: DNS - created ae.wikimedia.org for United Arab Emirates User Group wiki - T362529
• 21:02 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2209 (T361627)', diff saved to https://phabricator.wikimedia.org/P60824 and previous config saved to /var/cache/conftool/dbconfig/20240417-210256-marostegui.json
• 20:47 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2209', diff saved to https://phabricator.wikimedia.org/P60823 and previous config saved to /var/cache/conftool/dbconfig/20240417-204748-marostegui.json
• 20:44 eevans@deploy1002: helmfile [staging] DONE helmfile.d/services/echostore: apply
• 20:44 eevans@deploy1002: helmfile [staging] START helmfile.d/services/echostore: apply
• 20:44 cjming: end of UTC late backport window
• 20:44 eevans@deploy1002: helmfile [staging] DONE helmfile.d/services/sessionstore: apply
• 20:43 eevans@deploy1002: helmfile [staging] START helmfile.d/services/sessionstore: apply
• 20:43 cjming@deploy1002: Finished scap: Backport for Upstream tablet infobox styles (T3603861), Upstream tablet infobox styles (T3603861) (duration: 17m 30s)
• 20:32 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2209', diff saved to https://phabricator.wikimedia.org/P60822 and previous config saved to /var/cache/conftool/dbconfig/20240417-203241-marostegui.json
• 20:30 cjming@deploy1002: cjming and jdlrobson: Continuing with sync
• 20:29 cjming@deploy1002: cjming and jdlrobson: Backport for Upstream tablet infobox styles (T3603861), Upstream tablet infobox styles (T3603861) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:26 cjming@deploy1002: Started scap: Backport for Upstream tablet infobox styles (T3603861), Upstream tablet infobox styles (T3603861)
• 20:25 cjming@deploy1002: Finished scap: Backport for Enable WikimediaSkinStyles on English Wikipedia Vector 2022 skin (T362726), Enable night mode in AMC for all projects (T361555) (duration: 18m 13s)
• 20:17 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2209 (T361627)', diff saved to https://phabricator.wikimedia.org/P60821 and previous config saved to /var/cache/conftool/dbconfig/20240417-201733-marostegui.json
• 20:11 cjming@deploy1002: cjming and jdlrobson: Continuing with sync
• 20:09 cjming@deploy1002: cjming and jdlrobson: Backport for Enable WikimediaSkinStyles on English Wikipedia Vector 2022 skin (T362726), Enable night mode in AMC for all projects (T361555) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:06 cjming@deploy1002: Started scap: Backport for Enable WikimediaSkinStyles on English Wikipedia Vector 2022 skin (T362726), Enable night mode in AMC for all projects (T361555)
• 19:56 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2209 (T361627)', diff saved to https://phabricator.wikimedia.org/P60820 and previous config saved to /var/cache/conftool/dbconfig/20240417-195628-marostegui.json
• 19:56 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2209.codfw.wmnet with reason: Maintenance
• 19:56 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db2209.codfw.wmnet with reason: Maintenance
• 19:56 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2194 (T361627)', diff saved to https://phabricator.wikimedia.org/P60819 and previous config saved to /var/cache/conftool/dbconfig/20240417-195605-marostegui.json
• 19:46 eileen: civicrm upgraded from fdd12ed1 to 28adb4da
• 19:40 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2194', diff saved to https://phabricator.wikimedia.org/P60818 and previous config saved to /var/cache/conftool/dbconfig/20240417-194058-marostegui.json
• 19:25 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2194', diff saved to https://phabricator.wikimedia.org/P60817 and previous config saved to /var/cache/conftool/dbconfig/20240417-192551-marostegui.json
• 19:10 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2194 (T361627)', diff saved to https://phabricator.wikimedia.org/P60816 and previous config saved to /var/cache/conftool/dbconfig/20240417-191043-marostegui.json
• 18:56 ejegg: payments-wiki upgraded from 72e3bf19 to fb0367a4
• 18:49 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2194 (T361627)', diff saved to https://phabricator.wikimedia.org/P60815 and previous config saved to /var/cache/conftool/dbconfig/20240417-184931-marostegui.json
• 18:49 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2194.codfw.wmnet with reason: Maintenance
• 18:49 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db2194.codfw.wmnet with reason: Maintenance
• 18:49 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2190 (T361627)', diff saved to https://phabricator.wikimedia.org/P60814 and previous config saved to /var/cache/conftool/dbconfig/20240417-184908-marostegui.json
• 18:35 dancy@deploy1002: rebuilt and synchronized wikiversions files: group1 wikis to 1.43.0-wmf.1 refs T361395
• 18:34 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2190', diff saved to https://phabricator.wikimedia.org/P60813 and previous config saved to /var/cache/conftool/dbconfig/20240417-183401-marostegui.json
• 18:18 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2190', diff saved to https://phabricator.wikimedia.org/P60812 and previous config saved to /var/cache/conftool/dbconfig/20240417-181854-marostegui.json
• 18:03 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2190 (T361627)', diff saved to https://phabricator.wikimedia.org/P60810 and previous config saved to /var/cache/conftool/dbconfig/20240417-180346-marostegui.json
• 17:59 ebernhardson@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 17:59 ebernhardson@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 17:57 ebernhardson@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 17:57 ebernhardson@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 17:42 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2190 (T361627)', diff saved to https://phabricator.wikimedia.org/P60809 and previous config saved to /var/cache/conftool/dbconfig/20240417-174233-marostegui.json
• 17:42 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2190.codfw.wmnet with reason: Maintenance
• 17:42 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db2190.codfw.wmnet with reason: Maintenance
• 17:42 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2177 (T361627)', diff saved to https://phabricator.wikimedia.org/P60808 and previous config saved to /var/cache/conftool/dbconfig/20240417-174210-marostegui.json
• 17:27 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P60807 and previous config saved to /var/cache/conftool/dbconfig/20240417-172702-marostegui.json
• 17:14 sukhe: running authdns-update for adding magru geo-resources/IPs: T346722
• 17:11 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P60805 and previous config saved to /var/cache/conftool/dbconfig/20240417-171154-marostegui.json
• 16:56 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2177 (T361627)', diff saved to https://phabricator.wikimedia.org/P60804 and previous config saved to /var/cache/conftool/dbconfig/20240417-165647-marostegui.json
• 16:56 topranks: running authdns-update to make magru dns records live T362421
• 16:47 btullis@deploy1002: helmfile [eqiad] DONE helmfile.d/services/page-analytics: apply
• 16:46 btullis@deploy1002: helmfile [eqiad] START helmfile.d/services/page-analytics: apply
• 16:45 btullis@deploy1002: helmfile [codfw] DONE helmfile.d/services/page-analytics: apply
• 16:45 btullis@deploy1002: helmfile [codfw] START helmfile.d/services/page-analytics: apply
• 16:45 btullis@deploy1002: helmfile [staging] DONE helmfile.d/services/page-analytics: apply
• 16:44 btullis@deploy1002: helmfile [staging] START helmfile.d/services/page-analytics: apply
• 16:39 btullis@deploy1002: helmfile [eqiad] DONE helmfile.d/services/media-analytics: apply
• 16:39 btullis@deploy1002: helmfile [eqiad] START helmfile.d/services/media-analytics: apply
• 16:39 btullis@deploy1002: helmfile [codfw] DONE helmfile.d/services/media-analytics: apply
• 16:39 btullis@deploy1002: helmfile [codfw] START helmfile.d/services/media-analytics: apply
• 16:38 btullis@deploy1002: helmfile [staging] DONE helmfile.d/services/media-analytics: apply
• 16:38 btullis@deploy1002: helmfile [staging] START helmfile.d/services/media-analytics: apply
• 16:36 cmooney@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 16:36 cmooney@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Adding first entries for magru IPs - cmooney@cumin1002"
• 16:35 cmooney@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Adding first entries for magru IPs - cmooney@cumin1002"
• 16:35 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2177 (T361627)', diff saved to https://phabricator.wikimedia.org/P60803 and previous config saved to /var/cache/conftool/dbconfig/20240417-163532-marostegui.json
• 16:35 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2177.codfw.wmnet with reason: Maintenance
• 16:35 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db2177.codfw.wmnet with reason: Maintenance
• 16:35 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2156 (T361627)', diff saved to https://phabricator.wikimedia.org/P60802 and previous config saved to /var/cache/conftool/dbconfig/20240417-163518-marostegui.json
• 16:35 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2127 (T360332)', diff saved to https://phabricator.wikimedia.org/P60801 and previous config saved to /var/cache/conftool/dbconfig/20240417-163506-arnaudb.json
• 16:30 cmooney@cumin1002: START - Cookbook sre.dns.netbox
• 16:29 cmooney@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 16:29 cmooney@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Adding first entries for magru IPs - cmooney@cumin1002"
• 16:29 cmooney@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Adding first entries for magru IPs - cmooney@cumin1002"
• 16:27 cmooney@cumin1002: START - Cookbook sre.dns.netbox
• 16:25 cmooney@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 16:24 cmooney@cumin1002: START - Cookbook sre.dns.netbox
• 16:20 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P60800 and previous config saved to /var/cache/conftool/dbconfig/20240417-162008-marostegui.json
• 16:19 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2127', diff saved to https://phabricator.wikimedia.org/P60799 and previous config saved to /var/cache/conftool/dbconfig/20240417-161958-arnaudb.json
• 16:18 cmooney@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 16:18 cmooney@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Adding first entries for magru IPs - cmooney@cumin1002"
• 16:17 cmooney@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Adding first entries for magru IPs - cmooney@cumin1002"
• 16:14 claime: restarted rsyslog on mw2412 - T357616
• 16:13 cmooney@cumin1002: START - Cookbook sre.dns.netbox
• 16:10 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2119.codfw.wmnet
• 16:10 arnaudb@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 16:09 cdanis: above conftool actions had no impact on production, no dbctl config commit was performed.
• 16:09 arnaudb@cumin1002: START - Cookbook sre.dns.netbox
• 16:08 cmooney@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 16:08 cmooney@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Adding first entries for magru IPs - cmooney@cumin1002"
• 16:07 cdanis@cumin1002: conftool action : set/host_ip=10.64.16.8; selector: name=db1211
• 16:07 cdanis@cumin1002: conftool action : set/host_ip=1.1.1.1; selector: name=db1211
• 16:06 cdanis@cumin1002: conftool action : set/host_ip=10.64.16.8; selector: name=db1211
• 16:06 cmooney@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Adding first entries for magru IPs - cmooney@cumin1002"
• 16:05 cdanis@cumin1002: conftool action : set/host_ip=69.69.69.69; selector: name=db1211
• 16:05 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P60798 and previous config saved to /var/cache/conftool/dbconfig/20240417-160501-marostegui.json
• 16:04 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2127', diff saved to https://phabricator.wikimedia.org/P60797 and previous config saved to /var/cache/conftool/dbconfig/20240417-160451-arnaudb.json
• 16:04 arnaudb@cumin1002: dbctl commit (dc=all): 'db2119 depool T358741', diff saved to https://phabricator.wikimedia.org/P60796 and previous config saved to /var/cache/conftool/dbconfig/20240417-160443-arnaudb.json
• 16:04 btullis@deploy1002: helmfile [eqiad] DONE helmfile.d/services/geo-analytics: apply
• 16:04 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1114.eqiad.wmnet,service=(cdn|ats-be)
• 16:04 btullis@deploy1002: helmfile [eqiad] START helmfile.d/services/geo-analytics: apply
• 16:04 btullis@deploy1002: helmfile [codfw] DONE helmfile.d/services/geo-analytics: apply
• 16:03 cmooney@cumin1002: START - Cookbook sre.dns.netbox
• 16:03 btullis@deploy1002: helmfile [codfw] START helmfile.d/services/geo-analytics: apply
• 16:03 btullis@deploy1002: helmfile [staging] DONE helmfile.d/services/geo-analytics: apply
• 16:03 btullis@deploy1002: helmfile [staging] START helmfile.d/services/geo-analytics: apply
• 16:02 arnaudb@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2119.codfw.wmnet
• 16:00 cmooney@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 16:00 cmooney@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Adding first entries for magru IPs - cmooney@cumin1002"
• 15:59 cmooney@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Adding first entries for magru IPs - cmooney@cumin1002"
• 15:57 cmooney@cumin1002: START - Cookbook sre.dns.netbox
• 15:53 cmooney@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 15:53 cmooney@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Adding first entries for magru IPs - cmooney@cumin1002"
• 15:52 cmooney@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Adding first entries for magru IPs - cmooney@cumin1002"
• 15:51 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1114.eqiad.wmnet with OS bullseye
• 15:50 cmooney@cumin1002: START - Cookbook sre.dns.netbox
• 15:45 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2120.codfw.wmnet
• 15:45 arnaudb@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 15:45 arnaudb@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2120.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
• 15:44 arnaudb@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2120.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
• 15:42 arnaudb@cumin1002: START - Cookbook sre.dns.netbox
• 15:40 topranks: merging patch and updating dns servers with new magru ranges T362421
• 15:35 arnaudb@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2120.codfw.wmnet
• 15:34 cmooney@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 15:34 cmooney@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Adding first entries for magru IPs - cmooney@cumin1002"
• 15:33 cmooney@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Adding first entries for magru IPs - cmooney@cumin1002"
• 15:32 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2127 (T360332)', diff saved to https://phabricator.wikimedia.org/P60795 and previous config saved to /var/cache/conftool/dbconfig/20240417-153238-arnaudb.json
• 15:31 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1114.eqiad.wmnet with reason: host reimage
• 15:31 cmooney@cumin1002: START - Cookbook sre.dns.netbox
• 15:30 topranks: making magru IPs live in netbox and generating DNS records with cookbook T362421
• 15:27 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1114.eqiad.wmnet with reason: host reimage
• 15:20 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2156 (T361627)', diff saved to https://phabricator.wikimedia.org/P60794 and previous config saved to /var/cache/conftool/dbconfig/20240417-152023-marostegui.json
• 15:18 arnaudb@cumin1002: dbctl commit (dc=all): 'db2120 depool T358741', diff saved to https://phabricator.wikimedia.org/P60793 and previous config saved to /var/cache/conftool/dbconfig/20240417-151811-arnaudb.json
• 15:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2127 (T360332)', diff saved to https://phabricator.wikimedia.org/P60792 and previous config saved to /var/cache/conftool/dbconfig/20240417-151653-arnaudb.json
• 15:16 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2127.codfw.wmnet with reason: Maintenance
• 15:16 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2127.codfw.wmnet with reason: Maintenance
• 15:13 elukey@deploy1002: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'.
• 15:12 elukey@deploy1002: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'.
• 15:09 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.dhcp (exit_code=99) for host cp1115.eqiad.wmnet
• 15:07 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp1114.eqiad.wmnet with OS bullseye
• 15:06 vgutierrez: repool ncredir2001
• 15:05 Lucas_WMDE: UTC afternoon backport+config window (belatedly) done
• 15:04 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ mwscript namespaceDupes mlwiki --fix # T362653: 0 pages to fix, 0 were resolvable; 82 links to fix, 82 were resolvable, 0 were deleted.
• 15:03 logmsgbot: lucaswerkmeister-wmde@deploy1002 Finished scap: Backport for mlwiki: create draft namespace (T362653) (duration: 32m 43s)
• 14:59 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2156 (T361627)', diff saved to https://phabricator.wikimedia.org/P60790 and previous config saved to /var/cache/conftool/dbconfig/20240417-145916-marostegui.json
• 14:59 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2186.codfw.wmnet with reason: Maintenance
• 14:58 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2186.codfw.wmnet with reason: Maintenance
• 14:58 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2156.codfw.wmnet with reason: Maintenance
• 14:58 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db2156.codfw.wmnet with reason: Maintenance
• 14:58 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2149 (T361627)', diff saved to https://phabricator.wikimedia.org/P60789 and previous config saved to /var/cache/conftool/dbconfig/20240417-145838-marostegui.json
• 14:51 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1218 (T352010)', diff saved to https://phabricator.wikimedia.org/P60788 and previous config saved to /var/cache/conftool/dbconfig/20240417-145136-ladsgroup.json
• 14:51 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1218.eqiad.wmnet with reason: Maintenance
• 14:51 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1218.eqiad.wmnet with reason: Maintenance
• 14:51 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1207 (T352010)', diff saved to https://phabricator.wikimedia.org/P60787 and previous config saved to /var/cache/conftool/dbconfig/20240417-145113-ladsgroup.json
• 14:50 logmsgbot: lucaswerkmeister-wmde@deploy1002 anzx and lucaswerkmeister-wmde: Continuing with sync
• 14:44 elukey@deploy1002: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'.
• 14:44 elukey@deploy1002: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'.
• 14:43 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P60786 and previous config saved to /var/cache/conftool/dbconfig/20240417-144330-marostegui.json
• 14:36 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P60785 and previous config saved to /var/cache/conftool/dbconfig/20240417-143606-ladsgroup.json
• 14:34 logmsgbot: lucaswerkmeister-wmde@deploy1002 anzx and lucaswerkmeister-wmde: Backport for mlwiki: create draft namespace (T362653) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 14:33 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-role (exit_code=0) for role: insetup::data_persistence
• 14:31 marostegui@cumin1002: dbctl commit (dc=all): 'db2120 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P60784 and previous config saved to /var/cache/conftool/dbconfig/20240417-143103-root.json
• 14:31 logmsgbot: lucaswerkmeister-wmde@deploy1002 Started scap: Backport for mlwiki: create draft namespace (T362653)
• 14:28 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P60783 and previous config saved to /var/cache/conftool/dbconfig/20240417-142823-marostegui.json
• 14:22 jforrester@deploy1002: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
• 14:22 sukhe@cumin1002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cp1114.eqiad.wmnet
• 14:21 sukhe@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp1114.eqiad.wmnet
• 14:21 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P60782 and previous config saved to /var/cache/conftool/dbconfig/20240417-142057-ladsgroup.json
• 14:20 jforrester@deploy1002: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
• 14:20 jforrester@deploy1002: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
• 14:20 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp1114.eqiad.wmnet,service=(cdn|ats-be)
• 14:20 sukhe: depool cp1114.eqiad.wmnet for PXE boot testing issues and downgrade NIC firmware: T350179
• 14:19 jforrester@deploy1002: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
• 14:19 jforrester@deploy1002: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
• 14:18 jforrester@deploy1002: helmfile [staging] START helmfile.d/services/wikifunctions: apply
• 14:15 marostegui@cumin1002: dbctl commit (dc=all): 'db2120 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P60781 and previous config saved to /var/cache/conftool/dbconfig/20240417-141557-root.json
• 14:15 jforrester@deploy1002: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
• 14:13 jforrester@deploy1002: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
• 14:13 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2149 (T361627)', diff saved to https://phabricator.wikimedia.org/P60780 and previous config saved to /var/cache/conftool/dbconfig/20240417-141314-marostegui.json
• 14:13 jforrester@deploy1002: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
• 14:10 jforrester@deploy1002: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
• 14:10 jforrester@deploy1002: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
• 14:09 jforrester@deploy1002: helmfile [staging] START helmfile.d/services/wikifunctions: apply
• 14:09 jforrester@deploy1002: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
• 14:09 jforrester@deploy1002: helmfile [staging] START helmfile.d/services/wikifunctions: apply
• 14:08 jmm@cumin2002: START - Cookbook sre.puppet.migrate-role for role: insetup::data_persistence
• 14:08 logmsgbot: lucaswerkmeister-wmde@deploy1002 Finished scap: Backport for Revert "WikimediaEvents: Set IPoid URL and enable ip_reputation/score" (duration: 16m 49s)
• 14:06 vgutierrez: depool ncredir2001
• 14:05 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1207 (T352010)', diff saved to https://phabricator.wikimedia.org/P60779 and previous config saved to /var/cache/conftool/dbconfig/20240417-140549-ladsgroup.json
• 14:02 sukhe: running authdns-update for adding magru to geo-maps: T346722
• 14:00 marostegui@cumin1002: dbctl commit (dc=all): 'db2120 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P60778 and previous config saved to /var/cache/conftool/dbconfig/20240417-140051-root.json
• 13:55 logmsgbot: lucaswerkmeister-wmde@deploy1002 trainbranchbot and lucaswerkmeister-wmde: Continuing with sync
• 13:55 logmsgbot: lucaswerkmeister-wmde@deploy1002 trainbranchbot and lucaswerkmeister-wmde: Backport for Revert "WikimediaEvents: Set IPoid URL and enable ip_reputation/score" synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 13:52 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2149 (T361627)', diff saved to https://phabricator.wikimedia.org/P60777 and previous config saved to /var/cache/conftool/dbconfig/20240417-135253-marostegui.json
• 13:52 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2149.codfw.wmnet with reason: Maintenance
• 13:52 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db2149.codfw.wmnet with reason: Maintenance
• 13:51 logmsgbot: lucaswerkmeister-wmde@deploy1002 Started scap: Backport for Revert "WikimediaEvents: Set IPoid URL and enable ip_reputation/score"
• 13:49 logmsgbot: lucaswerkmeister-wmde@deploy1002 Sync cancelled.
• 13:45 marostegui@cumin1002: dbctl commit (dc=all): 'db2120 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P60776 and previous config saved to /var/cache/conftool/dbconfig/20240417-134545-root.json
• 13:40 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es1033.eqiad.wmnet
• 13:36 sukhe: running authdns-update for CR 1020823
• 13:36 logmsgbot: lucaswerkmeister-wmde@deploy1002 kharlan and lucaswerkmeister-wmde: Backport for WikimediaEvents: Set IPoid URL and enable ip_reputation/score (T354597) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 13:33 logmsgbot: lucaswerkmeister-wmde@deploy1002 Started scap: Backport for WikimediaEvents: Set IPoid URL and enable ip_reputation/score (T354597)
• 13:33 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2139.codfw.wmnet with reason: Maintenance
• 13:33 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db2139.codfw.wmnet with reason: Maintenance
• 13:33 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2127 (T361627)', diff saved to https://phabricator.wikimedia.org/P60775 and previous config saved to /var/cache/conftool/dbconfig/20240417-133318-marostegui.json
• 13:32 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1115.eqiad.wmnet,service=(cdn|ats-be)
• 13:30 marostegui@cumin1002: dbctl commit (dc=all): 'db2120 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P60774 and previous config saved to /var/cache/conftool/dbconfig/20240417-133040-root.json
• 13:29 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es1033.eqiad.wmnet
• 13:29 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es1026.eqiad.wmnet
• 13:23 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp1115.eqiad.wmnet
• 13:23 sukhe@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp1115.eqiad.wmnet
• 13:18 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2127', diff saved to https://phabricator.wikimedia.org/P60773 and previous config saved to /var/cache/conftool/dbconfig/20240417-131811-marostegui.json
• 13:18 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es1026.eqiad.wmnet
• 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es2033.codfw.wmnet
• 13:15 marostegui@cumin1002: dbctl commit (dc=all): 'db2120 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P60772 and previous config saved to /var/cache/conftool/dbconfig/20240417-131533-root.json
• 13:11 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es2033.codfw.wmnet
• 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es2031.codfw.wmnet
• 13:05 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2120.codfw.wmnet with OS bookworm
• 13:03 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2127', diff saved to https://phabricator.wikimedia.org/P60771 and previous config saved to /var/cache/conftool/dbconfig/20240417-130303-marostegui.json
• 13:01 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es2031.codfw.wmnet
• 13:00 marostegui@cumin1002: dbctl commit (dc=all): 'db2120 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P60770 and previous config saved to /var/cache/conftool/dbconfig/20240417-130027-root.json
• 12:54 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es2026.codfw.wmnet
• 12:47 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2127 (T361627)', diff saved to https://phabricator.wikimedia.org/P60769 and previous config saved to /var/cache/conftool/dbconfig/20240417-124756-marostegui.json
• 12:27 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2127 (T361627)', diff saved to https://phabricator.wikimedia.org/P60768 and previous config saved to /var/cache/conftool/dbconfig/20240417-122748-marostegui.json
• 12:27 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2127.codfw.wmnet with reason: Maintenance
• 12:27 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db2127.codfw.wmnet with reason: Maintenance
• 12:27 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2109 (T361627)', diff saved to https://phabricator.wikimedia.org/P60767 and previous config saved to /var/cache/conftool/dbconfig/20240417-122725-marostegui.json
• 12:25 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db2120.codfw.wmnet with OS bookworm
• 12:21 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2120', diff saved to https://phabricator.wikimedia.org/P60766 and previous config saved to /var/cache/conftool/dbconfig/20240417-122150-root.json
• 12:12 vgutierrez: repool ncredir2001
• 12:12 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P60765 and previous config saved to /var/cache/conftool/dbconfig/20240417-121218-marostegui.json
• 12:06 moritzm: upgrading PHP on mediawiki baremetal canaries servers T362511
• 11:57 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P60763 and previous config saved to /var/cache/conftool/dbconfig/20240417-115709-marostegui.json
• 11:57 stevemunene@deploy1002: helmfile [codfw] DONE helmfile.d/services/datahub: sync on main
• 11:46 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es2032.codfw.wmnet
• 11:44 vgutierrez: depool ncredir2001
• 11:42 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2109 (T361627)', diff saved to https://phabricator.wikimedia.org/P60762 and previous config saved to /var/cache/conftool/dbconfig/20240417-114201-marostegui.json
• 11:36 stevemunene@deploy1002: helmfile [codfw] START helmfile.d/services/datahub: apply on main
• 11:33 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es2032.codfw.wmnet
• 11:30 akosiaris@deploy1002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
• 11:30 akosiaris@deploy1002: helmfile [eqiad] START helmfile.d/admin 'apply'.
• 11:29 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es1032.eqiad.wmnet
• 11:29 akosiaris@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
• 11:24 akosiaris@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
• 11:24 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1167 (T352010)', diff saved to https://phabricator.wikimedia.org/P60761 and previous config saved to /var/cache/conftool/dbconfig/20240417-112418-ladsgroup.json
• 11:24 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
• 11:23 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
• 11:23 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1167.eqiad.wmnet with reason: Maintenance
• 11:23 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1167.eqiad.wmnet with reason: Maintenance
• 11:23 jiji@deploy1002: Finished scap: NoOp (duration: 09m 38s)
• 11:22 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es1032.eqiad.wmnet
• 11:20 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2109 (T361627)', diff saved to https://phabricator.wikimedia.org/P60760 and previous config saved to /var/cache/conftool/dbconfig/20240417-112040-marostegui.json
• 11:20 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2109.codfw.wmnet with reason: Maintenance
• 11:20 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db2109.codfw.wmnet with reason: Maintenance
• 11:20 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2105 (T361627)', diff saved to https://phabricator.wikimedia.org/P60759 and previous config saved to /var/cache/conftool/dbconfig/20240417-112017-marostegui.json
• 11:17 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es2030.codfw.wmnet
• 11:13 jiji@deploy1002: Started scap: NoOp
• 11:06 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es2030.codfw.wmnet
• 11:05 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P60758 and previous config saved to /var/cache/conftool/dbconfig/20240417-110510-marostegui.json
• 11:04 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
• 10:54 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
• 10:53 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
• 10:53 jiji@cumin1002: conftool action : set/pooled=true; selector: dnsdisc=mw-web-ro,name=eqiad
• 10:53 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/admin 'apply'.
• 10:50 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P60757 and previous config saved to /var/cache/conftool/dbconfig/20240417-105002-marostegui.json
• 10:46 jiji@cumin1002: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
• 10:45 effie: pool eqiad back for mw-web-ro, mw-api-int-ro and mw-api-ext-ro
• 10:44 jiji@cumin1002: conftool action : set/pooled=true; selector: dnsdisc=mw-api-int-ro,name=eqiad
• 10:42 jiji@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-wikifunctions: apply
• 10:42 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es1027.eqiad.wmnet
• 10:42 jiji@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-wikifunctions: apply
• 10:41 jiji@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
• 10:41 jiji@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
• 10:40 jiji@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
• 10:38 jiji@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
• 10:37 jiji@deploy1002: helmfile [eqiad] [main] DONE helmfile.d/services/mw-jobrunner : sync
• 10:36 jiji@deploy1002: helmfile [eqiad] [canary] DONE helmfile.d/services/mw-jobrunner : sync
• 10:36 jiji@deploy1002: helmfile [eqiad] [canary] START helmfile.d/services/mw-jobrunner : sync
• 10:36 jiji@deploy1002: helmfile [eqiad] [main] START helmfile.d/services/mw-jobrunner : sync
• 10:35 jiji@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
• 10:35 jiji@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
• 10:35 akosiaris@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
• 10:35 akosiaris@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
• 10:34 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2105 (T361627)', diff saved to https://phabricator.wikimedia.org/P60756 and previous config saved to /var/cache/conftool/dbconfig/20240417-103455-marostegui.json
• 10:34 akosiaris: apply the coredns patches for bumping instances from 4 to 6. They are noop, I am applying them to update helm's state.
• 10:34 jiji@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
• 10:34 akosiaris@deploy1002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
• 10:34 akosiaris@deploy1002: helmfile [eqiad] START helmfile.d/admin 'apply'.
• 10:34 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es1027.eqiad.wmnet
• 10:33 jiji@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
• 10:33 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es2028.codfw.wmnet
• 10:22 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es2028.codfw.wmnet
• 10:14 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2105 (T361627)', diff saved to https://phabricator.wikimedia.org/P60755 and previous config saved to /var/cache/conftool/dbconfig/20240417-101446-marostegui.json
• 10:14 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2105.codfw.wmnet with reason: Maintenance
• 10:14 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db2105.codfw.wmnet with reason: Maintenance
• 10:08 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: sync
• 10:08 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/services/wikifeeds: sync
• 10:06 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
• 10:06 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
• 10:00 akosiaris: manually bump coredns in eqiad to 6
• 09:59 akosiaris: manually bump coredns in codfw to 6
• 09:57 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1240.eqiad.wmnet with reason: Maintenance
• 09:57 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db1240.eqiad.wmnet with reason: Maintenance
• 09:57 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1223 (T361627)', diff saved to https://phabricator.wikimedia.org/P60753 and previous config saved to /var/cache/conftool/dbconfig/20240417-095731-marostegui.json
• 09:44 cgoubert@cumin1002: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
• 09:44 cgoubert@cumin1002: conftool action : set/pooled=false; selector: dnsdisc=mw-api-int-ro,name=eqiad
• 09:44 cgoubert@cumin1002: conftool action : set/pooled=false; selector: dnsdisc=mw-web-ro,name=eqiad
• 09:42 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P60750 and previous config saved to /var/cache/conftool/dbconfig/20240417-094223-marostegui.json
• 09:31 jiji@deploy1002: scap failed: KeyError 'production' (duration: 22m 21s)
• 09:29 marostegui@cumin1002: dbctl commit (dc=all): 'db2150 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P60749 and previous config saved to /var/cache/conftool/dbconfig/20240417-092923-root.json
• 09:27 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P60748 and previous config saved to /var/cache/conftool/dbconfig/20240417-092714-marostegui.json
• 09:14 marostegui@cumin1002: dbctl commit (dc=all): 'db2150 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P60747 and previous config saved to /var/cache/conftool/dbconfig/20240417-091418-root.json
• 09:12 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1223 (T361627)', diff saved to https://phabricator.wikimedia.org/P60746 and previous config saved to /var/cache/conftool/dbconfig/20240417-091203-marostegui.json
• 09:08 jiji@deploy1002: Started scap: Switch mediawiki in eqiad to use node-local mcrouter ds - T346690
• 09:05 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1223 (T361627)', diff saved to https://phabricator.wikimedia.org/P60745 and previous config saved to /var/cache/conftool/dbconfig/20240417-090539-marostegui.json
• 09:05 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1223.eqiad.wmnet with reason: Maintenance
• 09:05 jiji@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
• 09:05 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db1223.eqiad.wmnet with reason: Maintenance
• 09:05 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1212 (T361627)', diff saved to https://phabricator.wikimedia.org/P60744 and previous config saved to /var/cache/conftool/dbconfig/20240417-090516-marostegui.json
• 09:03 jiji@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
• 08:59 marostegui@cumin1002: dbctl commit (dc=all): 'db2150 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P60743 and previous config saved to /var/cache/conftool/dbconfig/20240417-085912-root.json
• 08:57 hashar@deploy1002: Finished scap: Backport for logging: pluralize $wmgDefaultMonologHandler (T238838) (duration: 16m 37s)
• 08:50 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P60742 and previous config saved to /var/cache/conftool/dbconfig/20240417-085009-marostegui.json
• 08:44 hashar@deploy1002: hashar: Continuing with sync
• 08:44 hashar@deploy1002: hashar: Backport for logging: pluralize $wmgDefaultMonologHandler (T238838) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 08:44 marostegui@cumin1002: dbctl commit (dc=all): 'db2150 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P60741 and previous config saved to /var/cache/conftool/dbconfig/20240417-084407-root.json
• 08:41 hashar@deploy1002: Started scap: Backport for logging: pluralize $wmgDefaultMonologHandler (T238838)
• 08:40 aqu: Deployed refinery using scap, then deployed onto hdfs
• 08:35 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P60739 and previous config saved to /var/cache/conftool/dbconfig/20240417-083501-marostegui.json
• 08:29 marostegui@cumin1002: dbctl commit (dc=all): 'db2150 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P60738 and previous config saved to /var/cache/conftool/dbconfig/20240417-082901-root.json
• 08:26 aqu@deploy1002: Finished deploy [analytics/refinery@c4e197f] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@c4e197fa] (duration: 02m 23s)
• 08:24 aqu@deploy1002: Started deploy [analytics/refinery@c4e197f] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@c4e197fa]
• 08:19 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1212 (T361627)', diff saved to https://phabricator.wikimedia.org/P60737 and previous config saved to /var/cache/conftool/dbconfig/20240417-081953-marostegui.json
• 08:16 aqu@deploy1002: Finished deploy [analytics/refinery@c4e197f] (thin): Regular analytics weekly train THIN [analytics/refinery@c4e197fa] (duration: 03m 39s)
• 08:13 marostegui@cumin1002: dbctl commit (dc=all): 'db2150 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P60736 and previous config saved to /var/cache/conftool/dbconfig/20240417-081356-root.json
• 08:13 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1212 (T361627)', diff saved to https://phabricator.wikimedia.org/P60735 and previous config saved to /var/cache/conftool/dbconfig/20240417-081326-marostegui.json
• 08:13 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
• 08:13 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
• 08:13 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1212.eqiad.wmnet with reason: Maintenance
• 08:13 aqu@deploy1002: Started deploy [analytics/refinery@c4e197f] (thin): Regular analytics weekly train THIN [analytics/refinery@c4e197fa]
• 08:13 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db1212.eqiad.wmnet with reason: Maintenance
• 08:12 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T361627)', diff saved to https://phabricator.wikimedia.org/P60734 and previous config saved to /var/cache/conftool/dbconfig/20240417-081256-marostegui.json
• 08:10 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kubestage2002.codfw.wmnet
• 08:07 aqu@deploy1002: Finished deploy [analytics/refinery@c4e197f]: Regular analytics weekly train [analytics/refinery@c4e197fa] (duration: 27m 57s)
• 08:03 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2150.codfw.wmnet with OS bookworm
• 08:00 jayme@cumin1002: START - Cookbook sre.hosts.reboot-single for host kubestage2002.codfw.wmnet
• 07:58 marostegui@cumin1002: dbctl commit (dc=all): 'db2150 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P60733 and previous config saved to /var/cache/conftool/dbconfig/20240417-075850-root.json
• 07:57 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P60732 and previous config saved to /var/cache/conftool/dbconfig/20240417-075748-marostegui.json
• 07:49 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db1173.eqiad.wmnet
• 07:42 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2150.codfw.wmnet with reason: host reimage
• 07:42 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P60731 and previous config saved to /var/cache/conftool/dbconfig/20240417-074241-marostegui.json
• 07:40 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db1173.eqiad.wmnet
• 07:39 aqu@deploy1002: Started deploy [analytics/refinery@c4e197f]: Regular analytics weekly train [analytics/refinery@c4e197fa]
• 07:39 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2150.codfw.wmnet with reason: host reimage
• 07:39 aqu: analytics/refinery deploy begin (added source jars 0.2.35)
• 07:38 jynus: restart db1216 database for mariadb upgrade
• 07:37 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db2214.codfw.wmnet
• 07:27 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T361627)', diff saved to https://phabricator.wikimedia.org/P60730 and previous config saved to /var/cache/conftool/dbconfig/20240417-072733-marostegui.json
• 07:27 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db2214.codfw.wmnet
• 07:26 jynus: restart db1240 database for mariadb upgrade
• 07:22 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db2150.codfw.wmnet with OS bookworm
• 07:21 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1198 (T361627)', diff saved to https://phabricator.wikimedia.org/P60729 and previous config saved to /var/cache/conftool/dbconfig/20240417-072122-marostegui.json
• 07:21 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2150', diff saved to https://phabricator.wikimedia.org/P60728 and previous config saved to /var/cache/conftool/dbconfig/20240417-072115-root.json
• 07:21 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1198.eqiad.wmnet with reason: Maintenance
• 07:21 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db1198.eqiad.wmnet with reason: Maintenance
• 07:21 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T361627)', diff saved to https://phabricator.wikimedia.org/P60727 and previous config saved to /var/cache/conftool/dbconfig/20240417-072059-marostegui.json
• 07:05 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P60726 and previous config saved to /var/cache/conftool/dbconfig/20240417-070552-marostegui.json
• 07:02 marostegui@cumin1002: dbctl commit (dc=all): 'db2182 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P60725 and previous config saved to /var/cache/conftool/dbconfig/20240417-070206-root.json
• 06:50 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P60724 and previous config saved to /var/cache/conftool/dbconfig/20240417-065044-marostegui.json
• 06:47 marostegui@cumin1002: dbctl commit (dc=all): 'db2182 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P60723 and previous config saved to /var/cache/conftool/dbconfig/20240417-064700-root.json
• 06:35 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T361627)', diff saved to https://phabricator.wikimedia.org/P60722 and previous config saved to /var/cache/conftool/dbconfig/20240417-063537-marostegui.json
• 06:31 marostegui@cumin1002: dbctl commit (dc=all): 'db2182 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P60721 and previous config saved to /var/cache/conftool/dbconfig/20240417-063155-root.json
• 06:29 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1189 (T361627)', diff saved to https://phabricator.wikimedia.org/P60720 and previous config saved to /var/cache/conftool/dbconfig/20240417-062918-marostegui.json
• 06:29 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1189.eqiad.wmnet with reason: Maintenance
• 06:29 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db1189.eqiad.wmnet with reason: Maintenance
• 06:28 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T361627)', diff saved to https://phabricator.wikimedia.org/P60719 and previous config saved to /var/cache/conftool/dbconfig/20240417-062856-marostegui.json
• 06:16 marostegui@cumin1002: dbctl commit (dc=all): 'db2182 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P60718 and previous config saved to /var/cache/conftool/dbconfig/20240417-061649-root.json
• 06:13 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P60717 and previous config saved to /var/cache/conftool/dbconfig/20240417-061349-marostegui.json
• 06:01 marostegui@cumin1002: dbctl commit (dc=all): 'db2182 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P60716 and previous config saved to /var/cache/conftool/dbconfig/20240417-060143-root.json
• 05:58 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P60715 and previous config saved to /var/cache/conftool/dbconfig/20240417-055841-marostegui.json
• 05:46 marostegui@cumin1002: dbctl commit (dc=all): 'db2182 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P60714 and previous config saved to /var/cache/conftool/dbconfig/20240417-054637-root.json
• 05:43 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T361627)', diff saved to https://phabricator.wikimedia.org/P60713 and previous config saved to /var/cache/conftool/dbconfig/20240417-054333-marostegui.json
• 05:37 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1175 (T361627)', diff saved to https://phabricator.wikimedia.org/P60712 and previous config saved to /var/cache/conftool/dbconfig/20240417-053716-marostegui.json
• 05:37 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance
• 05:36 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance
• 05:36 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T361627)', diff saved to https://phabricator.wikimedia.org/P60711 and previous config saved to /var/cache/conftool/dbconfig/20240417-053653-marostegui.json
• 05:35 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2182.codfw.wmnet with OS bookworm
• 05:31 marostegui@cumin1002: dbctl commit (dc=all): 'db2182 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P60710 and previous config saved to /var/cache/conftool/dbconfig/20240417-053131-root.json
• 05:26 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1207 (T352010)', diff saved to https://phabricator.wikimedia.org/P60709 and previous config saved to /var/cache/conftool/dbconfig/20240417-052600-ladsgroup.json
• 05:25 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1207.eqiad.wmnet with reason: Maintenance
• 05:25 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1207.eqiad.wmnet with reason: Maintenance
• 05:25 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1206 (T352010)', diff saved to https://phabricator.wikimedia.org/P60708 and previous config saved to /var/cache/conftool/dbconfig/20240417-052537-ladsgroup.json
• 05:21 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P60707 and previous config saved to /var/cache/conftool/dbconfig/20240417-052145-marostegui.json
• 05:15 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2182.codfw.wmnet with reason: host reimage
• 05:12 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2182.codfw.wmnet with reason: host reimage
• 05:10 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1206', diff saved to https://phabricator.wikimedia.org/P60706 and previous config saved to /var/cache/conftool/dbconfig/20240417-051029-ladsgroup.json
• 05:06 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P60705 and previous config saved to /var/cache/conftool/dbconfig/20240417-050638-marostegui.json
• 05:05 marostegui: Rename machine_vision tables on db1249 eqiad dbmaint s4 T362229
• 05:00 marostegui: dbmaint Upgrade s7 codfw to Bookworm and MariaDB 10.6 T362745
• 04:55 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1206', diff saved to https://phabricator.wikimedia.org/P60704 and previous config saved to /var/cache/conftool/dbconfig/20240417-045522-ladsgroup.json
• 04:55 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db2182.codfw.wmnet with OS bookworm
• 04:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2182', diff saved to https://phabricator.wikimedia.org/P60703 and previous config saved to /var/cache/conftool/dbconfig/20240417-045353-root.json
• 04:51 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T361627)', diff saved to https://phabricator.wikimedia.org/P60702 and previous config saved to /var/cache/conftool/dbconfig/20240417-045130-marostegui.json
• 04:45 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1166 (T361627)', diff saved to https://phabricator.wikimedia.org/P60701 and previous config saved to /var/cache/conftool/dbconfig/20240417-044517-marostegui.json
• 04:45 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
• 04:44 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
• 04:40 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1206 (T352010)', diff saved to https://phabricator.wikimedia.org/P60700 and previous config saved to /var/cache/conftool/dbconfig/20240417-044015-ladsgroup.json
• 04:39 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1222.eqiad.wmnet with reason: Maintenance
• 04:38 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db1222.eqiad.wmnet with reason: Maintenance
• 03:39 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1206 (T352010)', diff saved to https://phabricator.wikimedia.org/P60699 and previous config saved to /var/cache/conftool/dbconfig/20240417-033948-ladsgroup.json
• 03:39 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1206.eqiad.wmnet with reason: Maintenance
• 03:39 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1206.eqiad.wmnet with reason: Maintenance
• 03:39 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1196 (T352010)', diff saved to https://phabricator.wikimedia.org/P60698 and previous config saved to /var/cache/conftool/dbconfig/20240417-033926-ladsgroup.json
• 03:24 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P60697 and previous config saved to /var/cache/conftool/dbconfig/20240417-032418-ladsgroup.json
• 03:09 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P60696 and previous config saved to /var/cache/conftool/dbconfig/20240417-030911-ladsgroup.json
• 02:54 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1196 (T352010)', diff saved to https://phabricator.wikimedia.org/P60695 and previous config saved to /var/cache/conftool/dbconfig/20240417-025403-ladsgroup.json
• 02:48 ryankemper: T361525 Trying to powercycle `elastic2088` thru mgmt port (host not responding to ssh)
• 02:43 dani@deploy1002: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
• 02:43 dani@deploy1002: helmfile [codfw] START helmfile.d/services/miscweb: apply
• 02:43 dani@deploy1002: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
• 02:43 dani@deploy1002: helmfile [eqiad] START helmfile.d/services/miscweb: apply
• 02:43 dani@deploy1002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
• 02:42 dani@deploy1002: helmfile [staging] START helmfile.d/services/miscweb: apply

2024-04-16

• 23:25 hmonroy@deploy1002: Finished scap: Backport for [mediawikiwiki] enable CodeMirror V6 (T357795) (duration: 17m 29s)
• 23:12 hmonroy@deploy1002: musikanimal and hmonroy: Continuing with sync
• 23:11 hmonroy@deploy1002: musikanimal and hmonroy: Backport for [mediawikiwiki] enable CodeMirror V6 (T357795) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 23:08 hmonroy@deploy1002: Started scap: Backport for [mediawikiwiki] enable CodeMirror V6 (T357795)
• 23:06 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcontrol2009-dev.codfw.wmnet with OS bookworm
• 23:06 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002"
• 23:03 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002"
• 22:46 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcontrol2009-dev.codfw.wmnet with reason: host reimage
• 22:43 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcontrol2009-dev.codfw.wmnet with reason: host reimage
• 22:25 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host cloudcontrol2009-dev.codfw.wmnet with OS bookworm
• 21:54 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcontrol2009-dev.codfw.wmnet with OS bookworm
• 21:48 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 21:47 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 21:47 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 21:47 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 21:46 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 21:46 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 21:46 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 21:45 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 21:45 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 21:45 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 21:44 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 21:42 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 21:38 cjming: end of UTC late backport window
• 21:38 cjming@deploy1002: Finished scap: Backport for Use WikimediaMessages for template overrides (T361589) (duration: 19m 30s)
• 21:25 cjming@deploy1002: jdlrobson and cjming: Continuing with sync
• 21:21 cjming@deploy1002: jdlrobson and cjming: Backport for Use WikimediaMessages for template overrides (T361589) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 21:18 cjming@deploy1002: Started scap: Backport for Use WikimediaMessages for template overrides (T361589)
• 21:16 cjming@deploy1002: Finished scap: Backport for [phase 4] Vector-2022.js should no longer load legacy Vector site and user scripts/styles (T301212) (duration: 18m 26s)
• 21:02 cjming@deploy1002: cjming and jdlrobson: Continuing with sync
• 21:01 cjming@deploy1002: cjming and jdlrobson: Backport for [phase 4] Vector-2022.js should no longer load legacy Vector site and user scripts/styles (T301212) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:57 cjming@deploy1002: Started scap: Backport for [phase 4] Vector-2022.js should no longer load legacy Vector site and user scripts/styles (T301212)
• 20:56 cjming@deploy1002: Finished scap: Backport for Thumbnail styles generalized and moved to core (T360388) (duration: 22m 48s)
• 20:42 cjming@deploy1002: cjming and jdlrobson: Continuing with sync
• 20:36 cjming@deploy1002: cjming and jdlrobson: Backport for Thumbnail styles generalized and moved to core (T360388) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:33 cjming@deploy1002: Started scap: Backport for Thumbnail styles generalized and moved to core (T360388)
• 20:30 mutante: CI - jenkins and zuul-merger are re-enabled on contint1002 after distro upgrade to bullseye - T334517
• 20:26 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host cloudcontrol2009-dev.codfw.wmnet with OS bookworm
• 20:22 mutante: CI - re-enabled jenkins and zuul-merged on contint1002 after distro upgrade - T360964
• 20:22 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
• 20:22 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
• 20:22 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1246 (T361627)', diff saved to https://phabricator.wikimedia.org/P60693 and previous config saved to /var/cache/conftool/dbconfig/20240416-202206-marostegui.json
• 20:08 aqu: Weekly deploy of refinery using scap, then deployed onto hdfs
• 20:07 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1246', diff saved to https://phabricator.wikimedia.org/P60691 and previous config saved to /var/cache/conftool/dbconfig/20240416-200659-marostegui.json
• 19:55 pt1979@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cloudcontrol2009-dev']
• 19:53 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcontrol2009-dev']
• 19:51 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1246', diff saved to https://phabricator.wikimedia.org/P60690 and previous config saved to /var/cache/conftool/dbconfig/20240416-195151-marostegui.json
• 19:47 hashar@deploy1002: Finished deploy [zuul/deploy@efce3ee]: Redeploy Zuul following host reimaging - T334517 (duration: 00m 08s)
• 19:47 hashar@deploy1002: Started deploy [zuul/deploy@efce3ee]: Redeploy Zuul following host reimaging - T334517
• 19:42 aqu@deploy1002: Finished deploy [analytics/refinery@59f7d09] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@59f7d091] (duration: 02m 24s)
• 19:40 aqu@deploy1002: Started deploy [analytics/refinery@59f7d09] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@59f7d091]
• 19:38 aqu@deploy1002: Finished deploy [analytics/refinery@59f7d09] (thin): Regular analytics weekly train THIN [analytics/refinery@59f7d091] (duration: 04m 10s)
• 19:36 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1246 (T361627)', diff saved to https://phabricator.wikimedia.org/P60689 and previous config saved to /var/cache/conftool/dbconfig/20240416-193643-marostegui.json
• 19:33 aqu@deploy1002: Started deploy [analytics/refinery@59f7d09] (thin): Regular analytics weekly train THIN [analytics/refinery@59f7d091]
• 19:31 aqu@deploy1002: Finished deploy [analytics/refinery@59f7d09]: Regular analytics weekly train [analytics/refinery@59f7d091] (duration: 13m 08s)
• 19:24 aqu@deploy1002: Finished deploy [airflow-dags/analytics_test@9208108]: Regular analytics weekly train [airflow-dags/analytics_test@9208108e] (duration: 00m 10s)
• 19:23 aqu@deploy1002: Started deploy [airflow-dags/analytics_test@9208108]: Regular analytics weekly train [airflow-dags/analytics_test@9208108e]
• 19:18 aqu@deploy1002: Started deploy [analytics/refinery@59f7d09]: Regular analytics weekly train [analytics/refinery@59f7d091]
• 19:17 aqu: Deployment train for analytics/refinery
• 19:15 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1246 (T361627)', diff saved to https://phabricator.wikimedia.org/P60687 and previous config saved to /var/cache/conftool/dbconfig/20240416-191522-marostegui.json
• 19:15 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1246.eqiad.wmnet with reason: Maintenance
• 19:15 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db1246.eqiad.wmnet with reason: Maintenance
• 19:14 hashar@deploy1002: Finished deploy [zuul/deploy@efce3ee]: Redeploy Zuul following host reimaging - T334517 (duration: 00m 13s)
• 19:14 hashar@deploy1002: Started deploy [zuul/deploy@efce3ee]: Redeploy Zuul following host reimaging - T334517
• 19:12 hashar@deploy1002: Finished deploy [zuul/deploy@efce3ee]: Redeploy Zuul following host reimaging - T334517 (duration: 00m 03s)
• 19:12 hashar@deploy1002: Started deploy [zuul/deploy@efce3ee]: Redeploy Zuul following host reimaging - T334517
• 19:11 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1239.eqiad.wmnet with reason: Maintenance
• 19:11 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db1239.eqiad.wmnet with reason: Maintenance
• 19:11 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1233 (T361627)', diff saved to https://phabricator.wikimedia.org/P60686 and previous config saved to /var/cache/conftool/dbconfig/20240416-191128-marostegui.json
• 19:08 dancy@deploy1002: rebuilt and synchronized wikiversions files: group0 wikis to 1.43.0-wmf.1 refs T361395
• 18:56 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P60685 and previous config saved to /var/cache/conftool/dbconfig/20240416-185621-marostegui.json
• 18:52 aqu@deploy1002: Finished deploy [airflow-dags/analytics@9208108]: Regular analytics weekly train [airflow-dags/analytics@9208108e] (duration: 00m 26s)
• 18:52 aqu@deploy1002: Started deploy [airflow-dags/analytics@9208108]: Regular analytics weekly train [airflow-dags/analytics@9208108e]
• 18:50 dancy@deploy1002: Installation of scap version "4.77.0" completed for 340 hosts
• 18:49 dancy@deploy1002: Installing scap version "4.77.0" for 340 hosts
• 18:48 dancy@deploy1002: Finished scap: Backport for [Parser] Temporarily disable deprecation warnings for dynamic properties (T362692) (duration: 22m 56s)
• 18:44 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host contint1002.wikimedia.org with OS bullseye
• 18:41 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P60684 and previous config saved to /var/cache/conftool/dbconfig/20240416-184113-marostegui.json
• 18:40 mutante: contint1002 - sudo a2dismod mpm_event to work around known race condition and fix failed initial puppet run - T334517
• 18:35 dancy@deploy1002: cscott and dancy: Continuing with sync
• 18:29 bearloga@deploy1002: Finished deploy [airflow-dags/analytics_product@77af7cb]: (no justification provided) (duration: 00m 07s)
• 18:29 bearloga@deploy1002: Started deploy [airflow-dags/analytics_product@77af7cb]: (no justification provided)
• 18:29 dancy@deploy1002: cscott and dancy: Backport for [Parser] Temporarily disable deprecation warnings for dynamic properties (T362692) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 18:26 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1233 (T361627)', diff saved to https://phabricator.wikimedia.org/P60683 and previous config saved to /var/cache/conftool/dbconfig/20240416-182606-marostegui.json
• 18:26 dancy@deploy1002: Started scap: Backport for [Parser] Temporarily disable deprecation warnings for dynamic properties (T362692)
• 18:10 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1233 (T361627)', diff saved to https://phabricator.wikimedia.org/P60682 and previous config saved to /var/cache/conftool/dbconfig/20240416-181001-marostegui.json
• 18:09 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1233.eqiad.wmnet with reason: Maintenance
• 18:09 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db1233.eqiad.wmnet with reason: Maintenance
• 18:09 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1229 (T361627)', diff saved to https://phabricator.wikimedia.org/P60681 and previous config saved to /var/cache/conftool/dbconfig/20240416-180938-marostegui.json
• 18:07 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on contint1002.wikimedia.org with reason: host reimage
• 18:04 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on contint1002.wikimedia.org with reason: host reimage
• 17:54 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P60680 and previous config saved to /var/cache/conftool/dbconfig/20240416-175431-marostegui.json
• 17:52 dzahn@cumin2002: START - Cookbook sre.hosts.reimage for host contint1002.wikimedia.org with OS bullseye
• 17:51 mutante: CI - jenkins on contint1002 disabled - reimaging in progress
• 17:50 bearloga@deploy1002: Finished deploy [airflow-dags/analytics_product@bb33843]: (no justification provided) (duration: 00m 06s)
• 17:50 bearloga@deploy1002: Started deploy [airflow-dags/analytics_product@bb33843]: (no justification provided)
• 17:49 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on contint1002.wikimedia.org with reason: reimage https://phabricator.wikmedia.org/T334517
• 17:48 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on contint1002.wikimedia.org with reason: reimage https://phabricator.wikmedia.org/T334517
• 17:39 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P60679 and previous config saved to /var/cache/conftool/dbconfig/20240416-173923-marostegui.json
• 17:37 mutante: CI - disabling zuul-merger on contint1002 - there is another on contint2002
• 17:35 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcontrol2009-dev.mgmt.codfw.wmnet with reboot policy FORCED
• 17:25 marostegui@cumin1002: dbctl commit (dc=all): 'db2127 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P60678 and previous config saved to /var/cache/conftool/dbconfig/20240416-172515-root.json
• 17:24 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1229 (T361627)', diff saved to https://phabricator.wikimedia.org/P60677 and previous config saved to /var/cache/conftool/dbconfig/20240416-172415-marostegui.json
• 17:22 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1229 (T361627)', diff saved to https://phabricator.wikimedia.org/P60676 and previous config saved to /var/cache/conftool/dbconfig/20240416-172201-marostegui.json
• 17:21 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1229.eqiad.wmnet with reason: Maintenance
• 17:21 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db1229.eqiad.wmnet with reason: Maintenance
• 17:17 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1225.eqiad.wmnet with reason: Maintenance
• 17:17 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db1225.eqiad.wmnet with reason: Maintenance
• 17:17 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1197 (T361627)', diff saved to https://phabricator.wikimedia.org/P60675 and previous config saved to /var/cache/conftool/dbconfig/20240416-171738-marostegui.json
• 17:16 btullis@cumin1002: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop analytics cluster: Roll restart of jvm daemons for openjdk upgrade.
• 17:10 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1196 (T352010)', diff saved to https://phabricator.wikimedia.org/P60674 and previous config saved to /var/cache/conftool/dbconfig/20240416-171047-ladsgroup.json
• 17:10 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
• 17:10 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
• 17:10 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1196.eqiad.wmnet with reason: Maintenance
• 17:10 marostegui@cumin1002: dbctl commit (dc=all): 'db2127 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P60673 and previous config saved to /var/cache/conftool/dbconfig/20240416-171010-root.json
• 17:10 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1196.eqiad.wmnet with reason: Maintenance
• 17:10 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1186 (T352010)', diff saved to https://phabricator.wikimedia.org/P60672 and previous config saved to /var/cache/conftool/dbconfig/20240416-171006-ladsgroup.json
• 17:04 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host cloudcontrol2009-dev.mgmt.codfw.wmnet with reboot policy FORCED
• 17:02 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 17:02 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudcontrol2009 DNS add - pt1979@cumin2002"
• 17:02 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P60671 and previous config saved to /var/cache/conftool/dbconfig/20240416-170231-marostegui.json
• 17:01 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudcontrol2009 DNS add - pt1979@cumin2002"
• 16:59 pt1979@cumin2002: START - Cookbook sre.dns.netbox
• 16:55 marostegui@cumin1002: dbctl commit (dc=all): 'db2127 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P60670 and previous config saved to /var/cache/conftool/dbconfig/20240416-165504-root.json
• 16:54 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P60669 and previous config saved to /var/cache/conftool/dbconfig/20240416-165458-ladsgroup.json
• 16:47 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P60668 and previous config saved to /var/cache/conftool/dbconfig/20240416-164722-marostegui.json
• 16:39 marostegui@cumin1002: dbctl commit (dc=all): 'db2127 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P60666 and previous config saved to /var/cache/conftool/dbconfig/20240416-163958-root.json
• 16:39 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P60665 and previous config saved to /var/cache/conftool/dbconfig/20240416-163951-ladsgroup.json
• 16:32 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1197 (T361627)', diff saved to https://phabricator.wikimedia.org/P60664 and previous config saved to /var/cache/conftool/dbconfig/20240416-163215-marostegui.json
• 16:29 arnaudb@cumin1002: dbctl commit (dc=all): 'db2123 (re)pooling @ 100%: post maintenance repool', diff saved to https://phabricator.wikimedia.org/P60663 and previous config saved to /var/cache/conftool/dbconfig/20240416-162926-arnaudb.json
• 16:29 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1197 (T361627)', diff saved to https://phabricator.wikimedia.org/P60662 and previous config saved to /var/cache/conftool/dbconfig/20240416-162900-marostegui.json
• 16:28 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1197.eqiad.wmnet with reason: Maintenance
• 16:28 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db1197.eqiad.wmnet with reason: Maintenance
• 16:28 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1188 (T361627)', diff saved to https://phabricator.wikimedia.org/P60661 and previous config saved to /var/cache/conftool/dbconfig/20240416-162838-marostegui.json
• 16:24 marostegui@cumin1002: dbctl commit (dc=all): 'db2127 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P60660 and previous config saved to /var/cache/conftool/dbconfig/20240416-162452-root.json
• 16:24 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1186 (T352010)', diff saved to https://phabricator.wikimedia.org/P60659 and previous config saved to /var/cache/conftool/dbconfig/20240416-162443-ladsgroup.json
• 16:16 brennen: finished phabricator deploy for T362689 - believe things are currently stable
• 16:14 arnaudb@cumin1002: dbctl commit (dc=all): 'db2123 (re)pooling @ 75%: post maintenance repool', diff saved to https://phabricator.wikimedia.org/P60657 and previous config saved to /var/cache/conftool/dbconfig/20240416-161420-arnaudb.json
• 16:14 elukey@deploy1002: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'.
• 16:13 elukey@deploy1002: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'.
• 16:13 brennen@deploy1002: Finished deploy [phabricator/deployment@098b9c2]: deploy phab1004 for T362689 (duration: 00m 42s)
• 16:13 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P60656 and previous config saved to /var/cache/conftool/dbconfig/20240416-161330-marostegui.json
• 16:13 brennen@deploy1002: Started deploy [phabricator/deployment@098b9c2]: deploy phab1004 for T362689
• 16:13 rzl: rzl@mwmaint1002:~$ sudo systemctl start mediawiki_job_globalblocking-fixGlobalBlockWhitelist.service # T360516
• 16:12 brennen@deploy1002: Finished deploy [phabricator/deployment@098b9c2]: test deploy phab2002 for T362689 (duration: 00m 32s)
• 16:12 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db2217.codfw.wmnet
• 16:12 brennen@deploy1002: Started deploy [phabricator/deployment@098b9c2]: test deploy phab2002 for T362689
• 16:09 marostegui@cumin1002: dbctl commit (dc=all): 'db2127 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P60655 and previous config saved to /var/cache/conftool/dbconfig/20240416-160946-root.json
• 16:07 brennen: starting phabricator deploy for T362689
• 16:06 elukey@deploy1002: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'.
• 16:05 elukey@deploy1002: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'.
• 16:01 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on cp1115.eqiad.wmnet with reason: testing PXE boot issues
• 16:00 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on cp1115.eqiad.wmnet with reason: testing PXE boot issues
• 15:59 arnaudb@cumin1002: dbctl commit (dc=all): 'db2123 (re)pooling @ 50%: post maintenance repool', diff saved to https://phabricator.wikimedia.org/P60654 and previous config saved to /var/cache/conftool/dbconfig/20240416-155914-arnaudb.json
• 15:58 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db2217.codfw.wmnet
• 15:58 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P60653 and previous config saved to /var/cache/conftool/dbconfig/20240416-155823-marostegui.json
• 15:57 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2127.codfw.wmnet with OS bookworm
• 15:55 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db1231.eqiad.wmnet
• 15:54 marostegui@cumin1002: dbctl commit (dc=all): 'db2127 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P60652 and previous config saved to /var/cache/conftool/dbconfig/20240416-155440-root.json
• 15:49 arnaudb@cumin1002: dbctl commit (dc=all): 'db2214 (re)pooling @ 100%: Post clone', diff saved to https://phabricator.wikimedia.org/P60651 and previous config saved to /var/cache/conftool/dbconfig/20240416-154915-arnaudb.json
• 15:48 ebernhardson@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 15:47 ebernhardson@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 15:46 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db1231.eqiad.wmnet
• 15:46 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db1168.eqiad.wmnet
• 15:44 arnaudb@cumin1002: dbctl commit (dc=all): 'db2123 (re)pooling @ 25%: post maintenance repool', diff saved to https://phabricator.wikimedia.org/P60650 and previous config saved to /var/cache/conftool/dbconfig/20240416-154408-arnaudb.json
• 15:43 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1188 (T361627)', diff saved to https://phabricator.wikimedia.org/P60649 and previous config saved to /var/cache/conftool/dbconfig/20240416-154316-marostegui.json
• 15:39 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1188 (T361627)', diff saved to https://phabricator.wikimedia.org/P60648 and previous config saved to /var/cache/conftool/dbconfig/20240416-153902-marostegui.json
• 15:38 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1188.eqiad.wmnet with reason: Maintenance
• 15:38 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db1188.eqiad.wmnet with reason: Maintenance
• 15:38 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T361627)', diff saved to https://phabricator.wikimedia.org/P60647 and previous config saved to /var/cache/conftool/dbconfig/20240416-153839-marostegui.json
• 15:34 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2127.codfw.wmnet with reason: host reimage
• 15:34 arnaudb@cumin1002: dbctl commit (dc=all): 'db2214 (re)pooling @ 75%: Post clone', diff saved to https://phabricator.wikimedia.org/P60646 and previous config saved to /var/cache/conftool/dbconfig/20240416-153408-arnaudb.json
• 15:32 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db1168.eqiad.wmnet
• 15:31 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2127.codfw.wmnet with reason: host reimage
• 15:31 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db2129.codfw.wmnet
• 15:29 arnaudb@cumin1002: dbctl commit (dc=all): 'db2123 (re)pooling @ 15%: post maintenance repool', diff saved to https://phabricator.wikimedia.org/P60645 and previous config saved to /var/cache/conftool/dbconfig/20240416-152902-arnaudb.json
• 15:23 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P60644 and previous config saved to /var/cache/conftool/dbconfig/20240416-152331-marostegui.json
• 15:19 arnaudb@cumin1002: dbctl commit (dc=all): 'db2214 (re)pooling @ 50%: Post clone', diff saved to https://phabricator.wikimedia.org/P60643 and previous config saved to /var/cache/conftool/dbconfig/20240416-151902-arnaudb.json
• 15:17 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db2129.codfw.wmnet
• 15:17 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2199.codfw.wmnet with reason: Maintenance
• 15:17 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2199.codfw.wmnet with reason: Maintenance
• 15:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1244 (T360332)', diff saved to https://phabricator.wikimedia.org/P60642 and previous config saved to /var/cache/conftool/dbconfig/20240416-151649-arnaudb.json
• 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db1224.eqiad.wmnet
• 15:13 arnaudb@cumin1002: dbctl commit (dc=all): 'db2123 (re)pooling @ 10%: post maintenance repool', diff saved to https://phabricator.wikimedia.org/P60641 and previous config saved to /var/cache/conftool/dbconfig/20240416-151357-arnaudb.json
• 15:13 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db2127.codfw.wmnet with OS bookworm
• 15:10 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2127 T362616', diff saved to https://phabricator.wikimedia.org/P60640 and previous config saved to /var/cache/conftool/dbconfig/20240416-151032-root.json
• 15:09 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db2205 to s3 primary T362616', diff saved to https://phabricator.wikimedia.org/P60639 and previous config saved to /var/cache/conftool/dbconfig/20240416-150933-root.json
• 15:08 marostegui: Starting s3 codfw failover from db2127 to db2205 - T362616
• 15:08 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P60638 and previous config saved to /var/cache/conftool/dbconfig/20240416-150824-marostegui.json
• 15:07 brennen@deploy1002: Finished deploy [phabricator/deployment@7773191]: deploy phab1004 for T362666 (duration: 00m 30s)
• 15:06 brennen@deploy1002: Started deploy [phabricator/deployment@7773191]: deploy phab1004 for T362666
• 15:06 brennen@deploy1002: Finished deploy [phabricator/deployment@7773191]: test deploy phab2002 for T362666 (duration: 00m 32s)
• 15:05 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db1224.eqiad.wmnet
• 15:05 brennen@deploy1002: Started deploy [phabricator/deployment@7773191]: test deploy phab2002 for T362666
• 15:03 arnaudb@cumin1002: dbctl commit (dc=all): 'db2214 (re)pooling @ 20%: Post clone', diff saved to https://phabricator.wikimedia.org/P60637 and previous config saved to /var/cache/conftool/dbconfig/20240416-150356-arnaudb.json
• 15:03 samtar@deploy1002: Finished scap: Backport for IS: Set Phonos to Inline Audio Player mode on test.wiki (duration: 17m 17s)
• 15:03 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab1004.eqiad.wmnet with reason: Phabricator/Phorge update
• 15:03 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 0:30:00 on phab1004.eqiad.wmnet with reason: Phabricator/Phorge update
• 15:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P60636 and previous config saved to /var/cache/conftool/dbconfig/20240416-150141-arnaudb.json
• 15:00 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db1201.eqiad.wmnet
• 14:58 arnaudb@cumin1002: dbctl commit (dc=all): 'db2123 (re)pooling @ 5%: post maintenance repool', diff saved to https://phabricator.wikimedia.org/P60635 and previous config saved to /var/cache/conftool/dbconfig/20240416-145851-arnaudb.json
• 14:53 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T361627)', diff saved to https://phabricator.wikimedia.org/P60634 and previous config saved to /var/cache/conftool/dbconfig/20240416-145316-marostegui.json
• 14:50 samtar@deploy1002: samtar: Continuing with sync
• 14:50 marostegui@cumin1002: dbctl commit (dc=all): 'Set db2205 with weight 0 T362616', diff saved to https://phabricator.wikimedia.org/P60633 and previous config saved to /var/cache/conftool/dbconfig/20240416-144957-root.json
• 14:49 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 26 hosts with reason: Primary switchover s3 T362616
• 14:49 samtar@deploy1002: samtar: Backport for IS: Set Phonos to Inline Audio Player mode on test.wiki synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 14:49 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on 26 hosts with reason: Primary switchover s3 T362616
• 14:48 arnaudb@cumin1002: dbctl commit (dc=all): 'db2214 (re)pooling @ 10%: Post clone', diff saved to https://phabricator.wikimedia.org/P60632 and previous config saved to /var/cache/conftool/dbconfig/20240416-144850-arnaudb.json
• 14:48 btullis@cumin1002: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop analytics cluster: Roll restart of jvm daemons for openjdk upgrade.
• 14:47 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db1201.eqiad.wmnet
• 14:47 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1182 (T361627)', diff saved to https://phabricator.wikimedia.org/P60631 and previous config saved to /var/cache/conftool/dbconfig/20240416-144727-marostegui.json
• 14:47 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1182.eqiad.wmnet with reason: Maintenance
• 14:47 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db1182.eqiad.wmnet with reason: Maintenance
• 14:47 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1162 (T361627)', diff saved to https://phabricator.wikimedia.org/P60630 and previous config saved to /var/cache/conftool/dbconfig/20240416-144704-marostegui.json
• 14:46 samtar@deploy1002: Started scap: Backport for IS: Set Phonos to Inline Audio Player mode on test.wiki
• 14:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P60629 and previous config saved to /var/cache/conftool/dbconfig/20240416-144634-arnaudb.json
• 14:45 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db2193.codfw.wmnet
• 14:43 arnaudb@cumin1002: dbctl commit (dc=all): 'db2123 (re)pooling @ 2%: post maintenance repool', diff saved to https://phabricator.wikimedia.org/P60628 and previous config saved to /var/cache/conftool/dbconfig/20240416-144346-arnaudb.json
• 14:43 taavi@deploy1002: Finished scap: Backport for Disallow changing email on Wikitech directly (T360883) (duration: 16m 24s)
• 14:36 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db2193.codfw.wmnet
• 14:36 vgutierrez: pool ncredir2002
• 14:33 vgutierrez: depool ncredir2002
• 14:32 vgutierrez: pool ncredir2001
• 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db1187.eqiad.wmnet
• 14:31 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P60627 and previous config saved to /var/cache/conftool/dbconfig/20240416-143157-marostegui.json
• 14:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1244 (T360332)', diff saved to https://phabricator.wikimedia.org/P60626 and previous config saved to /var/cache/conftool/dbconfig/20240416-143126-arnaudb.json
• 14:30 taavi@deploy1002: taavi: Continuing with sync
• 14:29 taavi@deploy1002: taavi: Backport for Disallow changing email on Wikitech directly (T360883) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 14:28 arnaudb@cumin1002: dbctl commit (dc=all): 'db2123 (re)pooling @ 1%: post maintenance repool', diff saved to https://phabricator.wikimedia.org/P60625 and previous config saved to /var/cache/conftool/dbconfig/20240416-142840-arnaudb.json
• 14:28 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1244 (T360332)', diff saved to https://phabricator.wikimedia.org/P60624 and previous config saved to /var/cache/conftool/dbconfig/20240416-142808-arnaudb.json
• 14:28 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1244.eqiad.wmnet with reason: Maintenance
• 14:27 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1244.eqiad.wmnet with reason: Maintenance
• 14:26 taavi@deploy1002: Started scap: Backport for Disallow changing email on Wikitech directly (T360883)
• 14:26 jiji@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply
• 14:26 jiji@deploy1002: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply
• 14:23 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db1187.eqiad.wmnet
• 14:22 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2123.codfw.wmnet with reason: Maintenance
• 14:22 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2123.codfw.wmnet with reason: Maintenance
• 14:21 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db2114.codfw.wmnet
• 14:19 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2123.codfw.wmnet with OS bookworm
• 14:16 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P60623 and previous config saved to /var/cache/conftool/dbconfig/20240416-141649-marostegui.json
• 14:06 logmsgbot: lucaswerkmeister-wmde@deploy1002 Finished scap: Backport for Restrict local uploads to uploader user group in azwiki (T360847) (duration: 35m 04s)
• 14:02 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db2114.codfw.wmnet
• 14:01 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1162 (T361627)', diff saved to https://phabricator.wikimedia.org/P60622 and previous config saved to /var/cache/conftool/dbconfig/20240416-140142-marostegui.json
• 13:59 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1162 (T361627)', diff saved to https://phabricator.wikimedia.org/P60621 and previous config saved to /var/cache/conftool/dbconfig/20240416-135928-marostegui.json
• 13:59 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1162.eqiad.wmnet with reason: Maintenance
• 13:59 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db1162.eqiad.wmnet with reason: Maintenance
• 13:59 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T361627)', diff saved to https://phabricator.wikimedia.org/P60620 and previous config saved to /var/cache/conftool/dbconfig/20240416-135906-marostegui.json
• 13:58 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2123.codfw.wmnet with reason: host reimage
• 13:56 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2123.codfw.wmnet with reason: host reimage
• 13:53 logmsgbot: lucaswerkmeister-wmde@deploy1002 nmw03 and lucaswerkmeister-wmde: Continuing with sync
• 13:44 jiji@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply
• 13:44 jiji@deploy1002: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply
• 13:44 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P60619 and previous config saved to /var/cache/conftool/dbconfig/20240416-134358-marostegui.json
• 13:43 jiji@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply
• 13:43 jiji@deploy1002: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply
• 13:38 arnaudb@cumin1002: START - Cookbook sre.hosts.reimage for host db2123.codfw.wmnet with OS bookworm
• 13:36 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2123.codfw.wmnet with reason: T360116
• 13:36 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2123.codfw.wmnet with reason: T360116
• 13:34 logmsgbot: lucaswerkmeister-wmde@deploy1002 nmw03 and lucaswerkmeister-wmde: Backport for Restrict local uploads to uploader user group in azwiki (T360847) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 13:33 vgutierrez: depool ncredir2001
• 13:31 logmsgbot: lucaswerkmeister-wmde@deploy1002 Started scap: Backport for Restrict local uploads to uploader user group in azwiki (T360847)
• 13:29 logmsgbot: lucaswerkmeister-wmde@deploy1002 Finished scap: Backport for Remove 'obsolete-tag' from $wgSignatureAllowedLintErrors on Polish Wikipedia (T362414) (duration: 18m 39s)
• 13:28 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P60618 and previous config saved to /var/cache/conftool/dbconfig/20240416-132851-marostegui.json
• 13:23 vgutierrez: pool ncredir2001
• 13:20 vgutierrez: depool ncredir2001
• 13:20 vgutierrez: pool ncredir1001
• 13:18 vgutierrez: depool ncredir1001
• 13:17 vgutierrez: pool ncredir2001
• 13:16 logmsgbot: lucaswerkmeister-wmde@deploy1002 msz2001 and lucaswerkmeister-wmde: Continuing with sync
• 13:14 logmsgbot: lucaswerkmeister-wmde@deploy1002 msz2001 and lucaswerkmeister-wmde: Backport for Remove 'obsolete-tag' from $wgSignatureAllowedLintErrors on Polish Wikipedia (T362414) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 13:13 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T361627)', diff saved to https://phabricator.wikimedia.org/P60617 and previous config saved to /var/cache/conftool/dbconfig/20240416-131344-marostegui.json
• 13:11 vgutierrez: depool ncredir2001
• 13:11 vgutierrez: pool ncredir1001
• 13:11 logmsgbot: lucaswerkmeister-wmde@deploy1002 Started scap: Backport for Remove 'obsolete-tag' from $wgSignatureAllowedLintErrors on Polish Wikipedia (T362414)
• 13:07 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1156 (T361627)', diff saved to https://phabricator.wikimedia.org/P60616 and previous config saved to /var/cache/conftool/dbconfig/20240416-130710-marostegui.json
• 13:07 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
• 13:07 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
• 13:06 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1156.eqiad.wmnet with reason: Maintenance
• 13:06 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db1156.eqiad.wmnet with reason: Maintenance
• 13:06 vgutierrez: depool ncredir1001
• 13:05 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2204.codfw.wmnet with reason: Maintenance
• 13:05 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db2204.codfw.wmnet with reason: Maintenance
• 13:01 jiji@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply
• 13:01 jiji@deploy1002: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply
• 12:57 jiji@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
• 12:55 jiji@deploy1002: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
• 12:21 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db2124.codfw.wmnet
• 12:13 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db2124.codfw.wmnet
• 12:12 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2207 (T361627)', diff saved to https://phabricator.wikimedia.org/P60615 and previous config saved to /var/cache/conftool/dbconfig/20240416-121211-marostegui.json
• 12:09 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db2180.codfw.wmnet
• 11:58 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db2180.codfw.wmnet
• 11:57 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db2151.codfw.wmnet
• 11:57 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P60613 and previous config saved to /var/cache/conftool/dbconfig/20240416-115703-marostegui.json
• 11:50 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db2151.codfw.wmnet
• 11:48 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db2169.codfw.wmnet
• 11:41 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P60612 and previous config saved to /var/cache/conftool/dbconfig/20240416-114155-marostegui.json
• 11:30 stevemunene@deploy1002: helmfile [codfw] DONE helmfile.d/services/datahub: sync on main
• 11:26 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2207 (T361627)', diff saved to https://phabricator.wikimedia.org/P60611 and previous config saved to /var/cache/conftool/dbconfig/20240416-112648-marostegui.json
• 11:21 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2207 (T361627)', diff saved to https://phabricator.wikimedia.org/P60610 and previous config saved to /var/cache/conftool/dbconfig/20240416-112134-marostegui.json
• 11:21 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2207.codfw.wmnet with reason: Maintenance
• 11:21 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db2207.codfw.wmnet with reason: Maintenance
• 11:19 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db2169.codfw.wmnet
• 11:16 jiji@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
• 11:16 jiji@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
• 11:16 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2197.codfw.wmnet with reason: Maintenance
• 11:16 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db2197.codfw.wmnet with reason: Maintenance
• 11:16 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2189 (T361627)', diff saved to https://phabricator.wikimedia.org/P60609 and previous config saved to /var/cache/conftool/dbconfig/20240416-111602-marostegui.json
• 11:08 stevemunene@deploy1002: helmfile [codfw] START helmfile.d/services/datahub: apply on main
• 11:07 hnowlan@puppetmaster1001: conftool action : set/pooled=yes; selector: name=restbase1042.eqiad.wmnet
• 11:00 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P60608 and previous config saved to /var/cache/conftool/dbconfig/20240416-110055-marostegui.json
• 10:58 hnowlan@puppetmaster1001: conftool action : set/pooled=no; selector: name=restbase1042.eqiad.wmnet
• 10:56 hnowlan: disabling puppet on A:restbase before switching to cfssl
• 10:55 jiji@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
• 10:55 jiji@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
• 10:45 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P60607 and previous config saved to /var/cache/conftool/dbconfig/20240416-104547-marostegui.json
• 10:30 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2189 (T361627)', diff saved to https://phabricator.wikimedia.org/P60605 and previous config saved to /var/cache/conftool/dbconfig/20240416-103040-marostegui.json
• 10:25 marostegui@cumin1002: dbctl commit (dc=all): 'db2105 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P60604 and previous config saved to /var/cache/conftool/dbconfig/20240416-102540-root.json
• 10:25 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2189 (T361627)', diff saved to https://phabricator.wikimedia.org/P60603 and previous config saved to /var/cache/conftool/dbconfig/20240416-102510-marostegui.json
• 10:25 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2189.codfw.wmnet with reason: Maintenance
• 10:24 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db2189.codfw.wmnet with reason: Maintenance
• 10:24 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T361627)', diff saved to https://phabricator.wikimedia.org/P60602 and previous config saved to /var/cache/conftool/dbconfig/20240416-102447-marostegui.json
• 10:20 moritzm: upgrading PHP on remaining mwdebug servers T362511
• 10:19 cmooney@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 10:17 cmooney@cumin1002: START - Cookbook sre.dns.netbox
• 10:15 jayme: updated rsyslog to 8.2404.0-1~bpo11+1 on all k8s nodes - T357616
• 10:13 jayme@deploy1002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
• 10:13 moritzm: uploaded PHP 7.4 1:7.4.33-1+0~20221108.73+debian10~1.gbpa00350a+wmf10u2+icu67u2 to buster-wikimedia/component/icu67 T362511
• 10:12 jayme@deploy1002: helmfile [eqiad] START helmfile.d/admin 'apply'.
• 10:12 jayme@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
• 10:10 jayme@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
• 10:10 jayme@deploy1002: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
• 10:10 marostegui@cumin1002: dbctl commit (dc=all): 'db2105 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P60601 and previous config saved to /var/cache/conftool/dbconfig/20240416-101034-root.json
• 10:09 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P60600 and previous config saved to /var/cache/conftool/dbconfig/20240416-100939-marostegui.json
• 10:09 jayme@deploy1002: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
• 10:09 jayme@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
• 10:08 jayme@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
• 10:08 jayme@deploy1002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 10:08 jayme@deploy1002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 10:08 jayme@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 10:08 jayme@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'.
• 10:08 jayme@deploy1002: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
• 10:07 jayme@deploy1002: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
• 09:56 hnowlan@deploy1002: helmfile [eqiad] [main] DONE helmfile.d/services/mw-jobrunner : sync
• 09:55 hnowlan@deploy1002: helmfile [eqiad] [canary] DONE helmfile.d/services/mw-jobrunner : sync
• 09:55 marostegui@cumin1002: dbctl commit (dc=all): 'db2105 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P60599 and previous config saved to /var/cache/conftool/dbconfig/20240416-095528-root.json
• 09:54 hnowlan@deploy1002: helmfile [eqiad] [canary] START helmfile.d/services/mw-jobrunner : sync
• 09:54 hnowlan@deploy1002: helmfile [eqiad] [main] START helmfile.d/services/mw-jobrunner : sync
• 09:54 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P60598 and previous config saved to /var/cache/conftool/dbconfig/20240416-095432-marostegui.json
• 09:49 hnowlan@deploy1002: helmfile [codfw] [main] DONE helmfile.d/services/mw-jobrunner : sync
• 09:48 hnowlan@deploy1002: helmfile [codfw] [canary] DONE helmfile.d/services/mw-jobrunner : sync
• 09:48 hnowlan@deploy1002: helmfile [codfw] [canary] START helmfile.d/services/mw-jobrunner : sync
• 09:48 hnowlan@deploy1002: helmfile [codfw] [main] START helmfile.d/services/mw-jobrunner : sync
• 09:40 marostegui@cumin1002: dbctl commit (dc=all): 'db2105 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P60597 and previous config saved to /var/cache/conftool/dbconfig/20240416-094023-root.json
• 09:39 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T361627)', diff saved to https://phabricator.wikimedia.org/P60596 and previous config saved to /var/cache/conftool/dbconfig/20240416-093924-marostegui.json
• 09:33 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2175 (T361627)', diff saved to https://phabricator.wikimedia.org/P60595 and previous config saved to /var/cache/conftool/dbconfig/20240416-093318-marostegui.json
• 09:33 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2175.codfw.wmnet with reason: Maintenance
• 09:33 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db2175.codfw.wmnet with reason: Maintenance
• 09:32 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2148 (T361627)', diff saved to https://phabricator.wikimedia.org/P60594 and previous config saved to /var/cache/conftool/dbconfig/20240416-093255-marostegui.json
• 09:31 arnaudb: Starting s5 codfw failover from db2123 to db2213 - T362614 (forgot to send it)
• 09:30 arnaudb@cumin1002: dbctl commit (dc=all): 'Depool db2123 T362614', diff saved to https://phabricator.wikimedia.org/P60593 and previous config saved to /var/cache/conftool/dbconfig/20240416-093041-arnaudb.json
• 09:28 arnaudb@cumin1002: dbctl commit (dc=all): 'Promote db2213 to s5 primary T362614', diff saved to https://phabricator.wikimedia.org/P60592 and previous config saved to /var/cache/conftool/dbconfig/20240416-092800-arnaudb.json
• 09:25 marostegui@cumin1002: dbctl commit (dc=all): 'db2105 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P60591 and previous config saved to /var/cache/conftool/dbconfig/20240416-092517-root.json
• 09:21 slyngshede@cumin1002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cloudidm2001-dev.codfw.wmnet
• 09:20 slyngshede@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudidm2001-dev.codfw.wmnet with OS bookworm
• 09:17 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P60590 and previous config saved to /var/cache/conftool/dbconfig/20240416-091747-marostegui.json
• 09:10 marostegui@cumin1002: dbctl commit (dc=all): 'db2105 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P60588 and previous config saved to /var/cache/conftool/dbconfig/20240416-091009-root.json
• 09:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Set db2213 with weight 0 T362614', diff saved to https://phabricator.wikimedia.org/P60587 and previous config saved to /var/cache/conftool/dbconfig/20240416-090755-arnaudb.json
• 09:07 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 26 hosts with reason: Primary switchover s5 T362614
• 09:07 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on 26 hosts with reason: Primary switchover s5 T362614
• 09:06 arnaudb@cumin1002: dbctl commit (dc=all): 'db1161 (re)pooling @ 100%: post reimage repool', diff saved to https://phabricator.wikimedia.org/P60586 and previous config saved to /var/cache/conftool/dbconfig/20240416-090625-arnaudb.json
• 09:05 slyngshede@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudidm2001-dev.codfw.wmnet with reason: host reimage
• 09:02 slyngshede@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudidm2001-dev.codfw.wmnet with reason: host reimage
• 09:02 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P60585 and previous config saved to /var/cache/conftool/dbconfig/20240416-090240-marostegui.json
• 08:59 jayme: updated rsyslog to 8.2404.0-1~bpo11+1 on wikikube eqiad - T357616
• 08:55 marostegui@cumin1002: dbctl commit (dc=all): 'db2105 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P60584 and previous config saved to /var/cache/conftool/dbconfig/20240416-085503-root.json
• 08:51 arnaudb@cumin1002: dbctl commit (dc=all): 'db1161 (re)pooling @ 75%: post reimage repool', diff saved to https://phabricator.wikimedia.org/P60583 and previous config saved to /var/cache/conftool/dbconfig/20240416-085120-arnaudb.json
• 08:48 slyngshede@cumin1002: START - Cookbook sre.hosts.reimage for host cloudidm2001-dev.codfw.wmnet with OS bookworm
• 08:47 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2148 (T361627)', diff saved to https://phabricator.wikimedia.org/P60582 and previous config saved to /var/cache/conftool/dbconfig/20240416-084733-marostegui.json
• 08:47 jayme: updated rsyslog to 8.2404.0-1~bpo11+1 on wikikube codfw - T357616
• 08:46 slyngshede@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cloudidm2001-dev.codfw.wmnet - slyngshede@cumin1002"
• 08:45 slyngshede@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cloudidm2001-dev.codfw.wmnet - slyngshede@cumin1002"
• 08:45 slyngshede@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cloudidm2001-dev.codfw.wmnet on all recursors
• 08:45 slyngshede@cumin1002: START - Cookbook sre.dns.wipe-cache cloudidm2001-dev.codfw.wmnet on all recursors
• 08:45 slyngshede@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 08:45 slyngshede@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cloudidm2001-dev.codfw.wmnet - slyngshede@cumin1002"
• 08:44 slyngshede@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cloudidm2001-dev.codfw.wmnet - slyngshede@cumin1002"
• 08:42 slyngshede@cumin1002: START - Cookbook sre.dns.netbox
• 08:42 slyngshede@cumin1002: START - Cookbook sre.ganeti.makevm for new host cloudidm2001-dev.codfw.wmnet
• 08:41 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2148 (T361627)', diff saved to https://phabricator.wikimedia.org/P60581 and previous config saved to /var/cache/conftool/dbconfig/20240416-084118-marostegui.json
• 08:41 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2148.codfw.wmnet with reason: Maintenance
• 08:41 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db2148.codfw.wmnet with reason: Maintenance
• 08:40 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2138 (T361627)', diff saved to https://phabricator.wikimedia.org/P60580 and previous config saved to /var/cache/conftool/dbconfig/20240416-084055-marostegui.json
• 08:36 arnaudb@cumin1002: dbctl commit (dc=all): 'db1161 (re)pooling @ 50%: post reimage repool', diff saved to https://phabricator.wikimedia.org/P60579 and previous config saved to /var/cache/conftool/dbconfig/20240416-083614-arnaudb.json
• 08:34 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2105.codfw.wmnet with OS bookworm
• 08:25 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2138', diff saved to https://phabricator.wikimedia.org/P60578 and previous config saved to /var/cache/conftool/dbconfig/20240416-082548-marostegui.json
• 08:21 arnaudb@cumin1002: dbctl commit (dc=all): 'db1161 (re)pooling @ 25%: post reimage repool', diff saved to https://phabricator.wikimedia.org/P60577 and previous config saved to /var/cache/conftool/dbconfig/20240416-082108-arnaudb.json
• 08:19 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1161.eqiad.wmnet with OS bookworm
• 08:13 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2105.codfw.wmnet with reason: host reimage
• 08:10 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2138', diff saved to https://phabricator.wikimedia.org/P60576 and previous config saved to /var/cache/conftool/dbconfig/20240416-081040-marostegui.json
• 08:09 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2105.codfw.wmnet with reason: host reimage
• 07:56 volans@cumin1002: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
• 07:56 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1161.eqiad.wmnet with reason: host reimage
• 07:56 volans@cumin1002: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
• 07:55 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2138 (T361627)', diff saved to https://phabricator.wikimedia.org/P60575 and previous config saved to /var/cache/conftool/dbconfig/20240416-075533-marostegui.json
• 07:54 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1161.eqiad.wmnet with reason: host reimage
• 07:52 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db2105.codfw.wmnet with OS bookworm
• 07:50 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2105', diff saved to https://phabricator.wikimedia.org/P60574 and previous config saved to /var/cache/conftool/dbconfig/20240416-075056-root.json
• 07:49 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2138 (T361627)', diff saved to https://phabricator.wikimedia.org/P60573 and previous config saved to /var/cache/conftool/dbconfig/20240416-074952-marostegui.json
• 07:49 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2138.codfw.wmnet with reason: Maintenance
• 07:49 volans@cumin1002: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
• 07:49 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db2138.codfw.wmnet with reason: Maintenance
• 07:49 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2126 (T361627)', diff saved to https://phabricator.wikimedia.org/P60572 and previous config saved to /var/cache/conftool/dbconfig/20240416-074928-marostegui.json
• 07:43 volans@cumin1002: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
• 07:43 volans@cumin1002: END (FAIL) - Cookbook sre.netbox.update-extras (exit_code=1) rolling restart_daemons on A:netbox-canary
• 07:40 volans@cumin1002: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
• 07:40 arnaudb@cumin1002: START - Cookbook sre.hosts.reimage for host db1161.eqiad.wmnet with OS bookworm
• 07:39 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db[1154,1161].eqiad.wmnet with reason: T360116
• 07:39 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 3:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db[1154,1161].eqiad.wmnet with reason: T360116
• 07:38 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1161.eqiad.wmnet with reason: T360116
• 07:38 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 3:00:00 on db1161.eqiad.wmnet with reason: T360116
• 07:35 arnaudb@cumin1002: dbctl commit (dc=all): 'db1161 depool T360116', diff saved to https://phabricator.wikimedia.org/P60571 and previous config saved to /var/cache/conftool/dbconfig/20240416-073521-arnaudb.json
• 07:34 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P60570 and previous config saved to /var/cache/conftool/dbconfig/20240416-073420-marostegui.json
• 07:19 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P60569 and previous config saved to /var/cache/conftool/dbconfig/20240416-071913-marostegui.json
• 07:16 marostegui@cumin1002: dbctl commit (dc=all): 'db2156 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P60568 and previous config saved to /var/cache/conftool/dbconfig/20240416-071611-root.json
• 07:04 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2126 (T361627)', diff saved to https://phabricator.wikimedia.org/P60567 and previous config saved to /var/cache/conftool/dbconfig/20240416-070405-marostegui.json
• 07:02 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2126 (T361627)', diff saved to https://phabricator.wikimedia.org/P60566 and previous config saved to /var/cache/conftool/dbconfig/20240416-070139-marostegui.json
• 07:02 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2187.codfw.wmnet with reason: Maintenance
• 07:01 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2187.codfw.wmnet with reason: Maintenance
• 07:01 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2126.codfw.wmnet with reason: Maintenance
• 07:01 marostegui@cumin1002: dbctl commit (dc=all): 'db2156 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P60565 and previous config saved to /var/cache/conftool/dbconfig/20240416-070105-root.json
• 07:01 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db2126.codfw.wmnet with reason: Maintenance
• 07:01 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2125 (T361627)', diff saved to https://phabricator.wikimedia.org/P60564 and previous config saved to /var/cache/conftool/dbconfig/20240416-070100-marostegui.json
• 06:46 marostegui@cumin1002: dbctl commit (dc=all): 'db2156 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P60563 and previous config saved to /var/cache/conftool/dbconfig/20240416-064559-root.json
• 06:45 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2125', diff saved to https://phabricator.wikimedia.org/P60562 and previous config saved to /var/cache/conftool/dbconfig/20240416-064552-marostegui.json
• 06:37 volans: upgraed spicerack to v8.5.0 on cumin1002
• 06:36 volans@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:05:00 on cumin2002.codfw.wmnet with reason: test spicerack v8.5.0
• 06:36 volans@cumin2002: START - Cookbook sre.hosts.downtime for 0:05:00 on cumin2002.codfw.wmnet with reason: test spicerack v8.5.0
• 06:30 marostegui@cumin1002: dbctl commit (dc=all): 'db2156 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P60561 and previous config saved to /var/cache/conftool/dbconfig/20240416-063053-root.json
• 06:30 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2125', diff saved to https://phabricator.wikimedia.org/P60560 and previous config saved to /var/cache/conftool/dbconfig/20240416-063045-marostegui.json
• 06:15 marostegui@cumin1002: dbctl commit (dc=all): 'db2156 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P60559 and previous config saved to /var/cache/conftool/dbconfig/20240416-061546-root.json
• 06:15 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2125 (T361627)', diff saved to https://phabricator.wikimedia.org/P60558 and previous config saved to /var/cache/conftool/dbconfig/20240416-061536-marostegui.json
• 06:08 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2125 (T361627)', diff saved to https://phabricator.wikimedia.org/P60557 and previous config saved to /var/cache/conftool/dbconfig/20240416-060826-marostegui.json
• 06:08 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2125.codfw.wmnet with reason: Maintenance
• 06:08 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db2125.codfw.wmnet with reason: Maintenance
• 06:08 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2107 (T361627)', diff saved to https://phabricator.wikimedia.org/P60556 and previous config saved to /var/cache/conftool/dbconfig/20240416-060803-marostegui.json
• 06:00 marostegui@cumin1002: dbctl commit (dc=all): 'db2156 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P60555 and previous config saved to /var/cache/conftool/dbconfig/20240416-060034-root.json
• 05:52 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2107', diff saved to https://phabricator.wikimedia.org/P60554 and previous config saved to /var/cache/conftool/dbconfig/20240416-055256-marostegui.json
• 05:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1186 (T352010)', diff saved to https://phabricator.wikimedia.org/P60553 and previous config saved to /var/cache/conftool/dbconfig/20240416-055237-ladsgroup.json
• 05:52 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1186.eqiad.wmnet with reason: Maintenance
• 05:52 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1186.eqiad.wmnet with reason: Maintenance
• 05:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1169 (T352010)', diff saved to https://phabricator.wikimedia.org/P60552 and previous config saved to /var/cache/conftool/dbconfig/20240416-055215-ladsgroup.json
• 05:49 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2156.codfw.wmnet with OS bookworm
• 05:45 marostegui@cumin1002: dbctl commit (dc=all): 'db2156 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P60551 and previous config saved to /var/cache/conftool/dbconfig/20240416-054528-root.json
• 05:37 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2107', diff saved to https://phabricator.wikimedia.org/P60550 and previous config saved to /var/cache/conftool/dbconfig/20240416-053749-marostegui.json
• 05:37 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P60549 and previous config saved to /var/cache/conftool/dbconfig/20240416-053706-ladsgroup.json
• 05:26 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2156.codfw.wmnet with reason: host reimage
• 05:24 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2156.codfw.wmnet with reason: host reimage
• 05:22 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2107 (T361627)', diff saved to https://phabricator.wikimedia.org/P60548 and previous config saved to /var/cache/conftool/dbconfig/20240416-052241-marostegui.json
• 05:21 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P60547 and previous config saved to /var/cache/conftool/dbconfig/20240416-052158-ladsgroup.json
• 05:17 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2107 (T361627)', diff saved to Unable to send diff to phaste and previous config saved to /var/cache/conftool/dbconfig/20240416-051623-marostegui.json
• 05:16 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2107.codfw.wmnet with reason: Maintenance
• 05:16 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db2107.codfw.wmnet with reason: Maintenance
• 05:11 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2097.codfw.wmnet with reason: Maintenance
• 05:11 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db2097.codfw.wmnet with reason: Maintenance
• 05:06 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1169 (T352010)', diff saved to https://phabricator.wikimedia.org/P60546 and previous config saved to /var/cache/conftool/dbconfig/20240416-050651-ladsgroup.json
• 05:04 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db2156.codfw.wmnet with OS bookworm
• 05:03 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2156', diff saved to https://phabricator.wikimedia.org/P60545 and previous config saved to /var/cache/conftool/dbconfig/20240416-050315-root.json
• 04:03 mwpresync@deploy1002: Finished scap: testwikis wikis to 1.43.0-wmf.1 refs T361395 (duration: 57m 31s)
• 03:05 mwpresync@deploy1002: Started scap: testwikis wikis to 1.43.0-wmf.1 refs T361395
• 03:03 mwpresync@deploy1002: Pruned MediaWiki: 1.42.0-wmf.24 (duration: 03m 11s)
• 02:51 ebernhardson@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 02:51 ebernhardson@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 02:42 ebernhardson@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 02:42 ebernhardson@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 02:38 ebernhardson@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 02:38 ebernhardson@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply

2024-04-15

• 23:35 eileen: civicrm upgraded from 0445bfaa to fdd12ed1
• 23:17 eileen: civicrm upgraded from 4d5a4fc3 to 0445bfaa
• 22:57 ebernhardson@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 22:57 ebernhardson@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 22:44 ebernhardson@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 22:44 ebernhardson@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 22:09 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 19 hosts with reason: T362508
• 22:09 bking@cumin2002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on 19 hosts with reason: T362508
• 21:48 ebernhardson@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 21:48 ebernhardson@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 21:45 ebernhardson@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 21:45 ebernhardson@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 21:44 ebernhardson@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 21:44 ebernhardson@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 21:38 ebernhardson@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 21:37 ebernhardson@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 21:30 ebernhardson@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 21:30 ebernhardson@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 21:14 ebernhardson@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 21:14 ebernhardson@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 21:01 kindrobot: closing the UTC late backport window
• 21:00 kindrobot@deploy1002: Finished scap: Backport for zhwikivoyage: Make RelatedArticles extension usable on zhwikivoyage (T361427) (duration: 18m 30s)
• 20:48 kindrobot@deploy1002: s8321414 and kindrobot: Continuing with sync
• 20:44 kindrobot@deploy1002: s8321414 and kindrobot: Backport for zhwikivoyage: Make RelatedArticles extension usable on zhwikivoyage (T361427) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:42 kindrobot@deploy1002: Started scap: Backport for zhwikivoyage: Make RelatedArticles extension usable on zhwikivoyage (T361427)
• 20:37 ebernhardson@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 20:37 ebernhardson@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 20:36 ebernhardson@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 20:36 ebernhardson@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 20:36 kindrobot@deploy1002: Finished scap: Backport for Enable desktop watchlist on beta cluster, clean up old references (T109277), Enable night mode on template namespace (duration: 17m 06s)
• 20:35 ebernhardson@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 20:34 ebernhardson@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 20:34 ebernhardson@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 20:34 ebernhardson@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 20:30 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
• 20:29 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
• 20:29 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1236 (T356166)', diff saved to https://phabricator.wikimedia.org/P60539 and previous config saved to /var/cache/conftool/dbconfig/20240415-202943-marostegui.json
• 20:24 kindrobot@deploy1002: jdlrobson and kindrobot: Continuing with sync
• 20:21 kindrobot@deploy1002: jdlrobson and kindrobot: Backport for Enable desktop watchlist on beta cluster, clean up old references (T109277), Enable night mode on template namespace synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:19 kindrobot@deploy1002: Started scap: Backport for Enable desktop watchlist on beta cluster, clean up old references (T109277), Enable night mode on template namespace
• 20:19 ebernhardson@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 20:19 ebernhardson@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 20:14 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1236', diff saved to https://phabricator.wikimedia.org/P60538 and previous config saved to /var/cache/conftool/dbconfig/20240415-201436-marostegui.json
• 20:13 ebernhardson@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 20:13 ebernhardson@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 20:12 ebernhardson@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 20:12 ebernhardson@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 20:06 ebernhardson@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 20:06 ebernhardson@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 20:05 kindrobot: staring UTC late backport window
• 19:59 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1236', diff saved to https://phabricator.wikimedia.org/P60537 and previous config saved to /var/cache/conftool/dbconfig/20240415-195928-marostegui.json
• 19:44 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1236 (T356166)', diff saved to https://phabricator.wikimedia.org/P60536 and previous config saved to /var/cache/conftool/dbconfig/20240415-194420-marostegui.json
• 19:39 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1169 (T352010)', diff saved to https://phabricator.wikimedia.org/P60535 and previous config saved to /var/cache/conftool/dbconfig/20240415-193921-ladsgroup.json
• 19:39 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1169.eqiad.wmnet with reason: Maintenance
• 19:39 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1169.eqiad.wmnet with reason: Maintenance
• 19:39 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1163 (T352010)', diff saved to https://phabricator.wikimedia.org/P60534 and previous config saved to /var/cache/conftool/dbconfig/20240415-193858-ladsgroup.json
• 19:23 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1163', diff saved to https://phabricator.wikimedia.org/P60533 and previous config saved to /var/cache/conftool/dbconfig/20240415-192350-ladsgroup.json
• 19:12 mutante: deleting unused kibana-next.svc records from DNS - T234854
• 19:08 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1163', diff saved to https://phabricator.wikimedia.org/P60532 and previous config saved to /var/cache/conftool/dbconfig/20240415-190842-ladsgroup.json
• 19:01 mutante: deleting unused cas-logstash.wikimedia.org from DNS
• 18:53 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1163 (T352010)', diff saved to https://phabricator.wikimedia.org/P60531 and previous config saved to /var/cache/conftool/dbconfig/20240415-185334-ladsgroup.json
• 18:51 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=ncredir1001.eqiad.wmnet,service=nginx
• 18:51 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=ncredir1001.eqiad.wmnet,service=nginx
• 18:50 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1236 (T356166)', diff saved to https://phabricator.wikimedia.org/P60530 and previous config saved to /var/cache/conftool/dbconfig/20240415-185008-marostegui.json
• 18:50 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1236.eqiad.wmnet with reason: Maintenance
• 18:49 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1236.eqiad.wmnet with reason: Maintenance
• 18:49 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1227 (T356166)', diff saved to https://phabricator.wikimedia.org/P60529 and previous config saved to /var/cache/conftool/dbconfig/20240415-184945-marostegui.json
• 18:45 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=ncredir1002.eqiad.wmnet,service=nginx
• 18:45 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=ncredir1001.eqiad.wmnet,service=nginx
• 18:36 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov1006.eqiad.wmnet with OS bullseye
• 18:34 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P60528 and previous config saved to /var/cache/conftool/dbconfig/20240415-183437-marostegui.json
• 18:34 eevans@deploy1002: helmfile [staging] DONE helmfile.d/services/sessionstore: apply
• 18:24 eevans@deploy1002: helmfile [staging] START helmfile.d/services/sessionstore: apply
• 18:22 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir1002.eqiad.wmnet with OS bullseye
• 18:19 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P60527 and previous config saved to /var/cache/conftool/dbconfig/20240415-181930-marostegui.json
• 18:15 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov1006.eqiad.wmnet with reason: host reimage
• 18:12 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov1006.eqiad.wmnet with reason: host reimage
• 18:04 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1227 (T356166)', diff saved to https://phabricator.wikimedia.org/P60526 and previous config saved to /var/cache/conftool/dbconfig/20240415-180422-marostegui.json
• 18:02 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir1002.eqiad.wmnet with reason: host reimage
• 18:00 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 18:00 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 17:59 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host dbprov1006.eqiad.wmnet with OS bullseye
• 17:58 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir1002.eqiad.wmnet with reason: host reimage
• 17:43 brett@cumin2002: START - Cookbook sre.hosts.reimage for host ncredir1002.eqiad.wmnet with OS bullseye
• 17:42 bking@cumin2002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: T361647 - bking@cumin2002
• 17:38 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir1001.eqiad.wmnet with OS bullseye
• 17:23 taavi@deploy1002: Finished scap: Backport for wmf-config: add private subnets for magru (T346722) (duration: 17m 21s)
• 17:13 jynus: stop db2139 dbs for upgrade T360751
• 17:10 taavi@deploy1002: taavi and sukhe: Continuing with sync
• 17:08 taavi@deploy1002: taavi and sukhe: Backport for wmf-config: add private subnets for magru (T346722) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 17:06 jynus@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2101.codfw.wmnet
• 17:06 jynus@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 17:06 jynus@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2101.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin2002"
• 17:06 taavi@deploy1002: Started scap: Backport for wmf-config: add private subnets for magru (T346722)
• 17:05 jynus@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2101.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin2002"
• 17:03 jynus@cumin2002: START - Cookbook sre.dns.netbox
• 16:57 jynus@cumin2002: START - Cookbook sre.hosts.decommission for hosts db2101.codfw.wmnet
• 16:32 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 16:32 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 16:30 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1227 (T356166)', diff saved to https://phabricator.wikimedia.org/P60524 and previous config saved to /var/cache/conftool/dbconfig/20240415-163011-marostegui.json
• 16:30 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1227.eqiad.wmnet with reason: Maintenance
• 16:29 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1227.eqiad.wmnet with reason: Maintenance
• 16:29 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1202 (T356166)', diff saved to https://phabricator.wikimedia.org/P60523 and previous config saved to /var/cache/conftool/dbconfig/20240415-162949-marostegui.json
• 16:28 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 16:28 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 16:23 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: T361647 - bking@cumin2002
• 16:19 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 16:19 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 16:17 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 16:17 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 16:14 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir1001.eqiad.wmnet with reason: host reimage
• 16:14 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P60522 and previous config saved to /var/cache/conftool/dbconfig/20240415-161441-marostegui.json
• 16:14 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 16:14 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 16:11 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir1001.eqiad.wmnet with reason: host reimage
• 16:10 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 16:10 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 16:08 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 16:08 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 16:05 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 16:05 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 16:03 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 16:03 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 16:02 brett@cumin2002: START - Cookbook sre.hosts.reimage for host ncredir1001.eqiad.wmnet with OS bullseye
• 16:01 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 16:01 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 15:59 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P60521 and previous config saved to /var/cache/conftool/dbconfig/20240415-155932-marostegui.json
• 15:58 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 15:58 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 15:56 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 15:56 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 15:55 brett@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ncredir1001.eqiad.wmnet with OS bullseye
• 15:55 brett@cumin2002: START - Cookbook sre.hosts.reimage for host ncredir1001.eqiad.wmnet with OS bullseye
• 15:53 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 15:53 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 15:51 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 15:51 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 15:50 eevans@deploy1002: helmfile [staging] DONE helmfile.d/services/sessionstore: apply
• 15:48 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 15:48 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 15:45 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 15:45 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 15:44 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1202 (T356166)', diff saved to https://phabricator.wikimedia.org/P60520 and previous config saved to /var/cache/conftool/dbconfig/20240415-154422-marostegui.json
• 15:40 eevans@deploy1002: helmfile [staging] START helmfile.d/services/sessionstore: apply
• 15:21 arnaudb@cumin1002: dbctl commit (dc=all): 'db2111 (re)pooling @ 100%: post reimage repool', diff saved to https://phabricator.wikimedia.org/P60519 and previous config saved to /var/cache/conftool/dbconfig/20240415-152132-arnaudb.json
• 15:19 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 15:18 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 15:15 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dbprov1006.eqiad.wmnet with OS bullseye
• 15:11 bking@cumin2002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: T361647 - bking@cumin2002
• 15:06 arnaudb@cumin1002: dbctl commit (dc=all): 'db2111 (re)pooling @ 75%: post reimage repool', diff saved to https://phabricator.wikimedia.org/P60518 and previous config saved to /var/cache/conftool/dbconfig/20240415-150626-arnaudb.json
• 15:04 dancy@deploy1002: Installation of scap version "4.76.0" completed for 340 hosts
• 15:04 Daimona: Running query for T362365#9710047
• 15:03 dancy@deploy1002: Installing scap version "4.76.0" for 340 hosts
• 15:03 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1202 (T356166)', diff saved to https://phabricator.wikimedia.org/P60517 and previous config saved to /var/cache/conftool/dbconfig/20240415-150257-marostegui.json
• 15:03 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1202.eqiad.wmnet with reason: Maintenance
• 15:02 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1202.eqiad.wmnet with reason: Maintenance
• 15:02 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1194 (T356166)', diff saved to https://phabricator.wikimedia.org/P60516 and previous config saved to /var/cache/conftool/dbconfig/20240415-150235-marostegui.json
• 14:52 Dreamy_Jazz: Afternoon backport window done
• 14:51 arnaudb@cumin1002: dbctl commit (dc=all): 'db2111 (re)pooling @ 50%: post reimage repool', diff saved to https://phabricator.wikimedia.org/P60515 and previous config saved to /var/cache/conftool/dbconfig/20240415-145120-arnaudb.json
• 14:50 dreamyjazz@deploy1002: Finished scap: Backport for Define 'useYear' as true for temp user serial mapping config (T349506) (duration: 16m 16s)
• 14:48 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: T361647 - bking@cumin2002
• 14:47 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P60514 and previous config saved to /var/cache/conftool/dbconfig/20240415-144725-marostegui.json
• 14:41 jynus: fixed grants for db2098
• 14:37 dreamyjazz@deploy1002: dreamyjazz: Continuing with sync
• 14:36 dreamyjazz@deploy1002: dreamyjazz: Backport for Define 'useYear' as true for temp user serial mapping config (T349506) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 14:36 arnaudb@cumin1002: dbctl commit (dc=all): 'db2111 (re)pooling @ 25%: post reimage repool', diff saved to https://phabricator.wikimedia.org/P60513 and previous config saved to /var/cache/conftool/dbconfig/20240415-143614-arnaudb.json
• 14:34 dreamyjazz@deploy1002: Started scap: Backport for Define 'useYear' as true for temp user serial mapping config (T349506)
• 14:32 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P60512 and previous config saved to /var/cache/conftool/dbconfig/20240415-143217-marostegui.json
• 14:31 urbanecm@deploy1002: Finished scap: Backport for Add wgAutoCreateTempUser configuration for production (T349506 T337090), Change mul deployment on beta to limited version (T356169) (duration: 30m 12s)
• 14:31 stevemunene@deploy1002: helmfile [staging] DONE helmfile.d/services/datahub: sync on main
• 14:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P{cp4051.ulsfo.wmnet,cp5030.eqsin.wmnet,cp5032.eqsin.wmnet} and A:cp
• 14:27 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2111.codfw.wmnet with OS bookworm
• 14:23 stevemunene@deploy1002: helmfile [staging] START helmfile.d/services/datahub: apply on main
• 14:21 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P{cp4051.ulsfo.wmnet,cp5030.eqsin.wmnet,cp5032.eqsin.wmnet} and A:cp
• 14:18 urbanecm@deploy1002: urbanecm and dreamyjazz and arthurtaylor: Continuing with sync
• 14:17 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1194 (T356166)', diff saved to https://phabricator.wikimedia.org/P60511 and previous config saved to /var/cache/conftool/dbconfig/20240415-141710-marostegui.json
• 14:16 elukey: move cassandra instances on cassandra-dev to pki - T352647
• 14:14 urbanecm@deploy1002: urbanecm and dreamyjazz and arthurtaylor: Backport for Add wgAutoCreateTempUser configuration for production (T349506 T337090), Change mul deployment on beta to limited version (T356169) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 14:13 vgutierrez: uploaded tcp-mss-clamper 0.4+deb11u2 to bullseye-wikimedia (apt.wm.o)
• 14:09 bking@cumin2002: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: T361647 - bking@cumin2002
• 14:06 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2111.codfw.wmnet with reason: host reimage
• 14:04 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: T361647 - bking@cumin2002
• 14:04 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2111.codfw.wmnet with reason: host reimage
• 14:04 bking@cumin2002: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster relforge: T361647 - bking@cumin2002
• 14:04 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster relforge: T361647 - bking@cumin2002
• 14:01 urbanecm@deploy1002: Started scap: Backport for Add wgAutoCreateTempUser configuration for production (T349506 T337090), Change mul deployment on beta to limited version (T356169)
• 13:59 jynus: update dbprov2005 dbbackups password T362509
• 13:58 urbanecm@deploy1002: sync-world aborted: Backport for Add wgAutoCreateTempUser configuration for production (T349506 T337090), Change mul deployment on beta to limited version (T356169) (duration: 52m 11s)
• 13:54 TheresNoTime: `[samtar@mwmaint1002 ~]$ mwscript extensions/Flow/maintenance/FlowFixInconsistentBoards.php --wiki=zhwiki --namespaceName User_talk` T362530
• 13:54 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host dbprov1006.eqiad.wmnet with OS bullseye
• 13:48 arnaudb@cumin1002: START - Cookbook sre.hosts.reimage for host db2111.codfw.wmnet with OS bookworm
• 13:47 arnaudb@cumin1002: dbctl commit (dc=all): 'db2111 depool', diff saved to https://phabricator.wikimedia.org/P60510 and previous config saved to /var/cache/conftool/dbconfig/20240415-134710-arnaudb.json
• 13:46 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2111.codfw.wmnet with reason: reboot multiinstance replica
• 13:46 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on db2111.codfw.wmnet with reason: reboot multiinstance replica
• 13:45 arnaudb@cumin1002: dbctl commit (dc=all): 'db2128 (re)pooling @ 100%: post reimage repool', diff saved to https://phabricator.wikimedia.org/P60509 and previous config saved to /var/cache/conftool/dbconfig/20240415-134522-arnaudb.json
• 13:45 vgutierrez: update thirdparty/haproxy28 to 2.8.9 for bullseye-wikimedia (apt.wm.o)
• 13:37 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-esams and not P{cp3066.esams.wmnet,cp3069.esams.wmnet,cp3070.esams.wmnet,cp3071.esams.wmnet,cp3072.esams.wmnet,cp3073.esams.wmnet} and A:cp
• 13:34 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1194 (T356166)', diff saved to https://phabricator.wikimedia.org/P60508 and previous config saved to /var/cache/conftool/dbconfig/20240415-133433-marostegui.json
• 13:34 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1194.eqiad.wmnet with reason: Maintenance
• 13:34 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1194.eqiad.wmnet with reason: Maintenance
• 13:34 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1191 (T356166)', diff saved to https://phabricator.wikimedia.org/P60507 and previous config saved to /var/cache/conftool/dbconfig/20240415-133410-marostegui.json
• 13:30 arnaudb@cumin1002: dbctl commit (dc=all): 'db2128 (re)pooling @ 75%: post reimage repool', diff saved to https://phabricator.wikimedia.org/P60506 and previous config saved to /var/cache/conftool/dbconfig/20240415-133016-arnaudb.json
• 13:19 volans: upgraed spicerack to v8.5.0 on cumin2002
• 13:19 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P60505 and previous config saved to /var/cache/conftool/dbconfig/20240415-131902-marostegui.json
• 13:16 pt1979@cumin2002: START - Cookbook sre.hosts.dhcp for host cp1115.eqiad.wmnet
• 13:15 arnaudb@cumin1002: dbctl commit (dc=all): 'db2128 (re)pooling @ 50%: post reimage repool', diff saved to https://phabricator.wikimedia.org/P60504 and previous config saved to /var/cache/conftool/dbconfig/20240415-131510-arnaudb.json
• 13:08 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-esams and not P{cp3066.esams.wmnet,cp3069.esams.wmnet,cp3070.esams.wmnet,cp3071.esams.wmnet,cp3072.esams.wmnet,cp3073.esams.wmnet} and A:cp
• 13:08 urbanecm@deploy1002: urbanecm and arthurtaylor and dreamyjazz: Backport for Add wgAutoCreateTempUser configuration for production (T349506 T337090), Change mul deployment on beta to limited version (T356169) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 13:06 urbanecm@deploy1002: Started scap: Backport for Add wgAutoCreateTempUser configuration for production (T349506 T337090), Change mul deployment on beta to limited version (T356169)
• 13:03 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P60503 and previous config saved to /var/cache/conftool/dbconfig/20240415-130355-marostegui.json
• 13:00 arnaudb@cumin1002: dbctl commit (dc=all): 'db2128 (re)pooling @ 25%: post reimage repool', diff saved to https://phabricator.wikimedia.org/P60502 and previous config saved to /var/cache/conftool/dbconfig/20240415-130005-arnaudb.json
• 12:56 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2128.codfw.wmnet with OS bookworm
• 12:48 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1191 (T356166)', diff saved to https://phabricator.wikimedia.org/P60501 and previous config saved to /var/cache/conftool/dbconfig/20240415-124848-marostegui.json
• 12:12 jynus: deploy new database grants for m1 <- dbbprov1005
• 12:09 arnaudb@cumin1002: START - Cookbook sre.hosts.reimage for host db2128.codfw.wmnet with OS bookworm
• 12:06 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1191 (T356166)', diff saved to https://phabricator.wikimedia.org/P60500 and previous config saved to /var/cache/conftool/dbconfig/20240415-120650-marostegui.json
• 12:06 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1191.eqiad.wmnet with reason: Maintenance
• 12:06 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1191.eqiad.wmnet with reason: Maintenance
• 12:06 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T356166)', diff saved to https://phabricator.wikimedia.org/P60499 and previous config saved to /var/cache/conftool/dbconfig/20240415-120627-marostegui.json
• 12:06 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2128.codfw.wmnet
• 12:04 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-eqiad and not P{cp1112.eqiad.wmnet,cp1113.eqiad.wmnet,cp1115.eqiad.wmnet} and A:cp
• 12:00 arnaudb@cumin1002: START - Cookbook sre.mysql.upgrade for db2128.codfw.wmnet
• 11:58 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db[2128,2186].codfw.wmnet with reason: upgrade db2128 T360116
• 11:58 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db[2128,2186].codfw.wmnet with reason: upgrade db2128 T360116
• 11:57 arnaudb@cumin1002: dbctl commit (dc=all): 'db2128 depool T360116', diff saved to https://phabricator.wikimedia.org/P60498 and previous config saved to /var/cache/conftool/dbconfig/20240415-115708-arnaudb.json
• 11:51 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P60497 and previous config saved to /var/cache/conftool/dbconfig/20240415-115118-marostegui.json
• 11:36 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P60496 and previous config saved to /var/cache/conftool/dbconfig/20240415-113610-marostegui.json
• 11:21 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T356166)', diff saved to https://phabricator.wikimedia.org/P60495 and previous config saved to /var/cache/conftool/dbconfig/20240415-112102-marostegui.json
• 11:13 volans: uploaded spicerack_8.5.0 to apt.wikimedia.org bullseye-wikimedia
• 11:07 moritzm: imported shellcheck 0.7.1-1~bpo10+1 to component/shellcheck T362518
• 11:03 btullis@cumin1002: END (FAIL) - Cookbook sre.hadoop.roll-restart-masters (exit_code=99) restart masters for Hadoop analytics cluster: Restart of jvm daemons.
• 10:46 btullis@cumin1002: START - Cookbook sre.hadoop.roll-restart-masters restart masters for Hadoop analytics cluster: Restart of jvm daemons.
• 10:38 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1174 (T356166)', diff saved to https://phabricator.wikimedia.org/P60494 and previous config saved to /var/cache/conftool/dbconfig/20240415-103853-marostegui.json
• 10:38 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1174.eqiad.wmnet with reason: Maintenance
• 10:38 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1174.eqiad.wmnet with reason: Maintenance
• 10:33 Dreamy_Jazz: Restarting MediaModeration scanning script - https://wikitech.wikimedia.org/wiki/MediaModeration
• 10:31 godog: bounce prometheus@k8s-staging in eqiad - T343529
• 10:31 moritzm: imported lilypond/lilypond-data 2.22.0-10~bpo10+1 to component/lilypond T362518
• 10:22 claime: Launching build-base-images on build2001 - T362518
• 10:10 hashar@deploy1002: Finished deploy [gerrit/gerrit@47eacb9]: Update Javascript plugins for Gerrit 3.8 - T354886 (duration: 00m 07s)
• 10:10 hashar@deploy1002: Started deploy [gerrit/gerrit@47eacb9]: Update Javascript plugins for Gerrit 3.8 - T354886
• 09:57 hashar@deploy1002: Finished deploy [gerrit/gerrit@2f3d3d4]: Gerrit to 3.8.5 on gerrit1003 - T354886 (duration: 00m 06s)
• 09:56 hashar@deploy1002: Started deploy [gerrit/gerrit@2f3d3d4]: Gerrit to 3.8.5 on gerrit1003 - T354886
• 09:53 hashar@deploy1002: Finished deploy [gerrit/gerrit@2f3d3d4]: Gerrit to 3.8.5 on gerrit2002 - T354886 (duration: 00m 08s)
• 09:52 hashar@deploy1002: Started deploy [gerrit/gerrit@2f3d3d4]: Gerrit to 3.8.5 on gerrit2002 - T354886
• 09:50 btullis@cumin1002: END (PASS) - Cookbook sre.hadoop.roll-restart-masters (exit_code=0) restart masters for Hadoop test cluster: Restart of jvm daemons.
• 09:26 cgoubert@deploy1002: Finished scap: T351237 (duration: 11m 43s)
• 09:14 cgoubert@deploy1002: Started scap: T351237
• 09:12 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1171.eqiad.wmnet with reason: Maintenance
• 09:11 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1171.eqiad.wmnet with reason: Maintenance
• 09:11 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1170 (T356166)', diff saved to https://phabricator.wikimedia.org/P60493 and previous config saved to /var/cache/conftool/dbconfig/20240415-091145-marostegui.json
• 09:09 ladsgroup@deploy1002: Finished scap: Backport for Set all wikis to read new for pagelinks migration except trwiki, zhwiki (T351237) (duration: 08m 51s)
• 09:08 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1163 (T352010)', diff saved to https://phabricator.wikimedia.org/P60492 and previous config saved to /var/cache/conftool/dbconfig/20240415-090834-ladsgroup.json
• 09:08 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1163.eqiad.wmnet with reason: Maintenance
• 09:08 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1163.eqiad.wmnet with reason: Maintenance
• 09:03 ladsgroup@deploy1002: ladsgroup: Continuing with sync
• 09:02 ladsgroup@deploy1002: ladsgroup: Backport for Set all wikis to read new for pagelinks migration except trwiki, zhwiki (T351237) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 09:00 ladsgroup@deploy1002: Started scap: Backport for Set all wikis to read new for pagelinks migration except trwiki, zhwiki (T351237)
• 08:56 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P60491 and previous config saved to /var/cache/conftool/dbconfig/20240415-085638-marostegui.json
• 08:54 btullis@cumin1002: START - Cookbook sre.hadoop.roll-restart-masters restart masters for Hadoop test cluster: Restart of jvm daemons.
• 08:53 jynus: restart dbprov2005
• 08:46 godog: logstash.w.o now uses sso - T246998
• 08:45 btullis@cumin1002: END (PASS) - Cookbook sre.druid.roll-restart-workers (exit_code=0) for Druid test cluster: Roll restart of Druid jvm daemons.
• 08:41 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P60490 and previous config saved to /var/cache/conftool/dbconfig/20240415-084130-marostegui.json
• 08:35 btullis@cumin1002: START - Cookbook sre.druid.roll-restart-workers for Druid test cluster: Roll restart of Druid jvm daemons.
• 08:26 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1170 (T356166)', diff saved to https://phabricator.wikimedia.org/P60488 and previous config saved to /var/cache/conftool/dbconfig/20240415-082623-marostegui.json
• 08:01 Emperor: depool wdqs in codfw T362508
• 08:01 mvernon@cumin2002: conftool action : set/pooled=false; selector: dnsdisc=wdqs,name=codfw
• 07:55 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1217.eqiad.wmnet with reason: reboot multiinstance replica
• 07:54 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on db1217.eqiad.wmnet with reason: reboot multiinstance replica
• 07:48 jayme: restarting k8s-mlstaging and k8s-staging prometheus instances - T343529
• 07:11 dcausse: restarting blazegraph on wdqs1020 (BlazegraphFreeAllocatorsDecreasingRapidly)
• 06:57 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1170 (T356166)', diff saved to https://phabricator.wikimedia.org/P60487 and previous config saved to /var/cache/conftool/dbconfig/20240415-065659-marostegui.json
• 06:56 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1170.eqiad.wmnet with reason: Maintenance
• 06:56 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1170.eqiad.wmnet with reason: Maintenance
• 06:56 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T356166)', diff saved to https://phabricator.wikimedia.org/P60486 and previous config saved to /var/cache/conftool/dbconfig/20240415-065636-marostegui.json
• 06:41 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P60485 and previous config saved to /var/cache/conftool/dbconfig/20240415-064129-marostegui.json
• 06:26 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P60484 and previous config saved to /var/cache/conftool/dbconfig/20240415-062621-marostegui.json
• 06:11 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T356166)', diff saved to https://phabricator.wikimedia.org/P60483 and previous config saved to /var/cache/conftool/dbconfig/20240415-061114-marostegui.json
• 05:30 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1158 (T356166)', diff saved to https://phabricator.wikimedia.org/P60482 and previous config saved to /var/cache/conftool/dbconfig/20240415-053001-marostegui.json
• 05:29 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
• 05:29 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
• 05:29 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1158.eqiad.wmnet with reason: Maintenance
• 05:29 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1158.eqiad.wmnet with reason: Maintenance

2024-04-14

• 16:00 marostegui@cumin1002: dbctl commit (dc=all): 'Set db2142 as x2 codfw master', diff saved to https://phabricator.wikimedia.org/P60481 and previous config saved to /var/cache/conftool/dbconfig/20240414-160016-marostegui.json
• 11:22 marostegui: Restart x2 codfw master
• 11:17 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 6 hosts with reason: Investigating
• 11:17 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on 6 hosts with reason: Investigating

2024-04-13

• 23:39 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1244 (T356166)', diff saved to https://phabricator.wikimedia.org/P60479 and previous config saved to /var/cache/conftool/dbconfig/20240413-233953-marostegui.json
• 23:24 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P60478 and previous config saved to /var/cache/conftool/dbconfig/20240413-232443-marostegui.json
• 23:09 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P60477 and previous config saved to /var/cache/conftool/dbconfig/20240413-230935-marostegui.json
• 22:54 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1244 (T356166)', diff saved to https://phabricator.wikimedia.org/P60476 and previous config saved to /var/cache/conftool/dbconfig/20240413-225428-marostegui.json
• 15:42 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1244 (T356166)', diff saved to https://phabricator.wikimedia.org/P60475 and previous config saved to /var/cache/conftool/dbconfig/20240413-154240-marostegui.json
• 15:42 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1244.eqiad.wmnet with reason: Maintenance
• 15:42 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1244.eqiad.wmnet with reason: Maintenance
• 15:42 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1242 (T356166)', diff saved to https://phabricator.wikimedia.org/P60474 and previous config saved to /var/cache/conftool/dbconfig/20240413-154217-marostegui.json
• 15:27 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1242', diff saved to https://phabricator.wikimedia.org/P60473 and previous config saved to /var/cache/conftool/dbconfig/20240413-152709-marostegui.json
• 15:12 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1242', diff saved to https://phabricator.wikimedia.org/P60472 and previous config saved to /var/cache/conftool/dbconfig/20240413-151201-marostegui.json
• 14:56 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1242 (T356166)', diff saved to https://phabricator.wikimedia.org/P60471 and previous config saved to /var/cache/conftool/dbconfig/20240413-145653-marostegui.json
• 06:06 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1242 (T356166)', diff saved to https://phabricator.wikimedia.org/P60470 and previous config saved to /var/cache/conftool/dbconfig/20240413-060646-marostegui.json
• 06:06 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1242.eqiad.wmnet with reason: Maintenance
• 06:06 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1242.eqiad.wmnet with reason: Maintenance
• 00:52 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.dhcp (exit_code=99) for host cp1115.eqiad.wmnet

2024-04-12

• 21:03 cdanis@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
• 21:03 cdanis@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
• 20:43 cdanis@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
• 20:43 cdanis@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
• 19:36 bking@cumin2002: END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Unbanning all hosts in search_codfw
• 19:36 bking@cumin2002: START - Cookbook sre.elasticsearch.ban Unbanning all hosts in search_codfw
• 18:56 andrew@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts cloudbackup2002.codfw.wmnet
• 18:56 andrew@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 18:56 andrew@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudbackup2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1002"
• 18:55 andrew@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudbackup2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1002"
• 18:52 andrew@cumin1002: START - Cookbook sre.dns.netbox
• 18:47 andrew@cumin1002: START - Cookbook sre.hosts.decommission for hosts cloudbackup2002.codfw.wmnet
• 18:46 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudbackup2001.codfw.wmnet
• 18:46 andrew@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 18:46 andrew@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudbackup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1002"
• 18:44 andrew@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudbackup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1002"
• 18:40 andrew@cumin1002: START - Cookbook sre.dns.netbox
• 18:35 andrew@cumin1002: START - Cookbook sre.hosts.decommission for hosts cloudbackup2001.codfw.wmnet
• 17:00 mutante: crm2001 - on initial puppet run adding envoy build-envoy-config failed building config and service failed due to dependency issue. manual run of "sudo /usr/local/sbin/build-envoy-config -c /etc/envoy/" and restarted envoyproxy.service
• 16:19 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host matomo1003.eqiad.wmnet with OS bookworm
• 16:16 elukey: move cassandra instances on cassandra-dev to the new truststore (allowing PKI certs) - T352647
• 15:59 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
• 15:56 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on cp1115.eqiad.wmnet with reason: testing PXE boot issues
• 15:56 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on cp1115.eqiad.wmnet with reason: testing PXE boot issues
• 15:55 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'readability' for release 'main' .
• 15:53 isaranto@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
• 15:52 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on elastic2090.codfw.wmnet with reason: T353878
• 15:51 bking@cumin2002: START - Cookbook sre.hosts.downtime for 1:00:00 on elastic2090.codfw.wmnet with reason: T353878
• 15:51 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
• 15:50 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
• 15:50 bking@cumin2002: END (FAIL) - Cookbook sre.elasticsearch.ban (exit_code=99) Banning hosts: elastic2090 for reboot to get rid of broken systemd units - bking@cumin2002 - T353878
• 15:50 bking@cumin2002: START - Cookbook sre.elasticsearch.ban Banning hosts: elastic2090 for reboot to get rid of broken systemd units - bking@cumin2002 - T353878
• 15:50 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on matomo1003.eqiad.wmnet with reason: host reimage
• 15:49 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' .
• 15:49 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
• 15:48 pt1979@cumin2002: START - Cookbook sre.hosts.dhcp for host cp1115.eqiad.wmnet
• 15:47 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' .
• 15:46 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on matomo1003.eqiad.wmnet with reason: host reimage
• 15:46 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
• 15:32 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host matomo1003.eqiad.wmnet with OS bookworm
• 15:31 btullis@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host matomo1003.eqiad.wmnet with OS bookworm
• 15:23 ayounsi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "magru - ayounsi@cumin1002"
• 15:22 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' .
• 15:21 ayounsi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "magru - ayounsi@cumin1002"
• 15:07 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'article-descriptions' for release 'main' .
• 15:03 elukey@deploy1002: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'.
• 15:03 ayounsi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "magru - ayounsi@cumin1002"
• 15:03 elukey@deploy1002: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'.
• 15:02 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host matomo1003.eqiad.wmnet with OS bookworm
• 15:01 ayounsi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "magru - ayounsi@cumin1002"
• 14:59 btullis@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host matomo1003.eqiad.wmnet with OS bookworm
• 14:22 hashar@deploy1002: Finished scap: Backport for Parser::statelessFetchTemplate: don't add interwiki redirects to dependencies (T362221) (duration: 16m 29s)
• 14:19 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
• 14:18 elukey@deploy1002: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'.
• 14:18 elukey@deploy1002: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'.
• 14:17 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host matomo1003.eqiad.wmnet with OS bookworm
• 14:09 hashar@deploy1002: hashar and jforrester: Continuing with sync
• 14:08 hashar@deploy1002: hashar and jforrester: Backport for Parser::statelessFetchTemplate: don't add interwiki redirects to dependencies (T362221) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 14:08 sukhe: depool cp1115 for PXE boot issue testing: T350179
• 14:07 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp1115.eqiad.wmnet,service=(cdn|ats-be)
• 14:05 hashar@deploy1002: Started scap: Backport for Parser::statelessFetchTemplate: don't add interwiki redirects to dependencies (T362221)
• 12:53 jayme: updated rsyslog to 8.2404.0-1~bpo11+1 on staging-codfw and staging-eqiad k8s clusters - T357616
• 12:20 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1249', diff saved to https://phabricator.wikimedia.org/P60466 and previous config saved to /var/cache/conftool/dbconfig/20240412-122045-marostegui.json
• 12:05 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1249', diff saved to https://phabricator.wikimedia.org/P60464 and previous config saved to /var/cache/conftool/dbconfig/20240412-120537-marostegui.json
• 12:02 btullis@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host matomo1003.eqiad.wmnet with OS bookworm
• 11:50 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1249 (T356166)', diff saved to https://phabricator.wikimedia.org/P60463 and previous config saved to /var/cache/conftool/dbconfig/20240412-115029-marostegui.json
• 11:33 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host matomo1003.eqiad.wmnet with OS bookworm
• 11:06 jelto@cumin1002: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Upgrade GitLab to new version
• 10:55 urbanecm: mwmaint1002: mwscript extensions/GrowthExperiments/maintenance/fixLinkRecommendationData.php --wiki=frwiki --search-index (T362367)
• 09:58 urbanecm: mwmaint1002: mwscript extensions/GrowthExperiments/maintenance/fixLinkRecommendationData.php --wiki=eswiki --search-index (T362367)
• 09:36 moritzm: installing postgresql-common bugfix updates from Bullseye point release
• 09:26 moritzm: installing debootstrap bugfix updates from Bullseye point release
• 09:25 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on matomo1003.eqiad.wmnet with reason: Still in setup
• 09:25 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on matomo1003.eqiad.wmnet with reason: Still in setup
• 08:56 jelto@cumin1002: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Upgrade GitLab to new version
• 07:24 marostegui@cumin1002: dbctl commit (dc=all): 'db2109 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P60461 and previous config saved to /var/cache/conftool/dbconfig/20240412-072435-root.json
• 07:09 marostegui@cumin1002: dbctl commit (dc=all): 'db2109 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P60460 and previous config saved to /var/cache/conftool/dbconfig/20240412-070930-root.json
• 06:54 marostegui@cumin1002: dbctl commit (dc=all): 'db2109 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P60459 and previous config saved to /var/cache/conftool/dbconfig/20240412-065424-root.json
• 06:39 marostegui@cumin1002: dbctl commit (dc=all): 'db2109 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P60458 and previous config saved to /var/cache/conftool/dbconfig/20240412-063918-root.json
• 06:24 marostegui@cumin1002: dbctl commit (dc=all): 'db2109 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P60457 and previous config saved to /var/cache/conftool/dbconfig/20240412-062412-root.json
• 06:09 marostegui@cumin1002: dbctl commit (dc=all): 'db2109 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P60456 and previous config saved to /var/cache/conftool/dbconfig/20240412-060907-root.json
• 05:56 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2109.codfw.wmnet with OS bookworm
• 05:54 marostegui@cumin1002: dbctl commit (dc=all): 'db2109 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P60455 and previous config saved to /var/cache/conftool/dbconfig/20240412-055401-root.json
• 05:35 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2109.codfw.wmnet with reason: host reimage
• 05:33 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2109.codfw.wmnet with reason: host reimage
• 05:23 moritzm: prune obsolete nginx debs on apt-staging after switch to new nginx provider scheme T329529
• 05:17 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db2109.codfw.wmnet with OS bookworm
• 05:16 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2109', diff saved to https://phabricator.wikimedia.org/P60454 and previous config saved to /var/cache/conftool/dbconfig/20240412-051606-root.json
• 03:33 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1249 (T356166)', diff saved to https://phabricator.wikimedia.org/P60453 and previous config saved to /var/cache/conftool/dbconfig/20240412-033317-marostegui.json
• 03:33 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1249.eqiad.wmnet with reason: Maintenance
• 03:32 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1249.eqiad.wmnet with reason: Maintenance
• 03:32 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1248 (T356166)', diff saved to https://phabricator.wikimedia.org/P60452 and previous config saved to /var/cache/conftool/dbconfig/20240412-033254-marostegui.json
• 03:17 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1248', diff saved to https://phabricator.wikimedia.org/P60451 and previous config saved to /var/cache/conftool/dbconfig/20240412-031744-marostegui.json
• 03:02 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1248', diff saved to https://phabricator.wikimedia.org/P60450 and previous config saved to /var/cache/conftool/dbconfig/20240412-030237-marostegui.json
• 02:47 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1248 (T356166)', diff saved to https://phabricator.wikimedia.org/P60449 and previous config saved to /var/cache/conftool/dbconfig/20240412-024729-marostegui.json
• 01:05 denisse: Manually deleting /srv/syslog/.linux.dhcp.DictModel/syslog.log from November 30 on centrallog1002 and centrallog2002 after the prune_old_srv_syslog_directories.service failed to delete the non-empty directory - T362376

2024-04-11

• 23:04 cstone: civicrm upgraded from c2569254 to 4d5a4fc3
• 20:20 urbanecm@deploy1002: Finished scap: Backport for ext-EventLogging: Add mediawiki.product_metrics.wikifunctions_ui to $wgEventLoggingStreamNames (duration: 17m 38s)
• 20:08 urbanecm@deploy1002: urbanecm and phuedx: Continuing with sync
• 20:05 urbanecm@deploy1002: urbanecm and phuedx: Backport for ext-EventLogging: Add mediawiki.product_metrics.wikifunctions_ui to $wgEventLoggingStreamNames synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:03 urbanecm@deploy1002: Started scap: Backport for ext-EventLogging: Add mediawiki.product_metrics.wikifunctions_ui to $wgEventLoggingStreamNames
• 19:41 eevans@deploy1002: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
• 19:40 eevans@deploy1002: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
• 19:35 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1248 (T356166)', diff saved to https://phabricator.wikimedia.org/P60448 and previous config saved to /var/cache/conftool/dbconfig/20240411-193537-marostegui.json
• 19:35 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1248.eqiad.wmnet with reason: Maintenance
• 19:35 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1248.eqiad.wmnet with reason: Maintenance
• 19:35 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1247 (T356166)', diff saved to https://phabricator.wikimedia.org/P60447 and previous config saved to /var/cache/conftool/dbconfig/20240411-193514-marostegui.json
• 19:20 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1247', diff saved to https://phabricator.wikimedia.org/P60446 and previous config saved to /var/cache/conftool/dbconfig/20240411-192006-marostegui.json
• 19:04 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1247', diff saved to https://phabricator.wikimedia.org/P60445 and previous config saved to /var/cache/conftool/dbconfig/20240411-190459-marostegui.json
• 18:49 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1247 (T356166)', diff saved to https://phabricator.wikimedia.org/P60443 and previous config saved to /var/cache/conftool/dbconfig/20240411-184951-marostegui.json
• 17:50 dancy@deploy1002: rebuilt and synchronized wikiversions files: group2 wikis to 1.42.0-wmf.26 refs T360158
• 17:27 swfrench@deploy1002: Finished scap: (no justification provided) (duration: 07m 57s)
• 17:20 swfrench@deploy1002: Started scap: (no justification provided)
• 17:12 bd808@deploy1002: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
• 17:12 bd808@deploy1002: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
• 17:11 bd808@deploy1002: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
• 17:11 bd808@deploy1002: helmfile [codfw] START helmfile.d/services/developer-portal: apply
• 17:11 bd808@deploy1002: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
• 17:10 bd808@deploy1002: helmfile [staging] START helmfile.d/services/developer-portal: apply
• 16:45 hashar@deploy1002: Finished scap: Backport for Revert "Update mobile search for dark mode, remove unused functions in MobilePage.php" (T362297) (duration: 16m 47s)
• 16:33 hashar@deploy1002: hashar: Continuing with sync
• 16:31 hashar@deploy1002: hashar: Backport for Revert "Update mobile search for dark mode, remove unused functions in MobilePage.php" (T362297) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 16:29 hashar@deploy1002: Started scap: Backport for Revert "Update mobile search for dark mode, remove unused functions in MobilePage.php" (T362297)
• 16:27 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host matomo1003.eqiad.wmnet with OS bookworm
• 16:15 arnaudb@cumin1002: dbctl commit (dc=all): 'db2111 (re)pooling @ 100%: repool', diff saved to https://phabricator.wikimedia.org/P60442 and previous config saved to /var/cache/conftool/dbconfig/20240411-161536-arnaudb.json
• 16:15 arnaudb@cumin1002: dbctl commit (dc=all): 'db2110 (re)pooling @ 100%: repool', diff saved to https://phabricator.wikimedia.org/P60441 and previous config saved to /var/cache/conftool/dbconfig/20240411-161522-arnaudb.json
• 16:03 herron: beginning rolling hardware upgrades for titan100[12] T361251
• 16:00 arnaudb@cumin1002: dbctl commit (dc=all): 'db2111 (re)pooling @ 75%: repool', diff saved to https://phabricator.wikimedia.org/P60440 and previous config saved to /var/cache/conftool/dbconfig/20240411-160030-arnaudb.json
• 16:00 arnaudb@cumin1002: dbctl commit (dc=all): 'db2110 (re)pooling @ 75%: repool', diff saved to https://phabricator.wikimedia.org/P60439 and previous config saved to /var/cache/conftool/dbconfig/20240411-160016-arnaudb.json
• 15:58 marostegui@cumin1002: dbctl commit (dc=all): 'db2149 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P60438 and previous config saved to /var/cache/conftool/dbconfig/20240411-155836-root.json
• 15:56 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host moss-fe2002.codfw.wmnet with OS bookworm
• 15:51 mvernon@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host moss-fe1002.eqiad.wmnet with OS bookworm
• 15:47 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-drmrs and A:cp
• 15:45 btullis@cumin1002: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
• 15:45 arnaudb@cumin1002: dbctl commit (dc=all): 'db2111 (re)pooling @ 50%: repool', diff saved to https://phabricator.wikimedia.org/P60437 and previous config saved to /var/cache/conftool/dbconfig/20240411-154524-arnaudb.json
• 15:45 arnaudb@cumin1002: dbctl commit (dc=all): 'db2110 (re)pooling @ 50%: repool', diff saved to https://phabricator.wikimedia.org/P60436 and previous config saved to /var/cache/conftool/dbconfig/20240411-154510-arnaudb.json
• 15:43 marostegui@cumin1002: dbctl commit (dc=all): 'db2149 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P60435 and previous config saved to /var/cache/conftool/dbconfig/20240411-154330-root.json
• 15:39 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on moss-fe2002.codfw.wmnet with reason: host reimage
• 15:36 mvernon@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on moss-fe2002.codfw.wmnet with reason: host reimage
• 15:35 mvernon@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on moss-fe1002.eqiad.wmnet with reason: host reimage
• 15:33 btullis@cumin1002: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
• 15:31 mvernon@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on moss-fe1002.eqiad.wmnet with reason: host reimage
• 15:30 arnaudb@cumin1002: dbctl commit (dc=all): 'db2111 (re)pooling @ 25%: repool', diff saved to https://phabricator.wikimedia.org/P60434 and previous config saved to /var/cache/conftool/dbconfig/20240411-153019-arnaudb.json
• 15:30 arnaudb@cumin1002: dbctl commit (dc=all): 'db2110 (re)pooling @ 25%: repool', diff saved to https://phabricator.wikimedia.org/P60433 and previous config saved to /var/cache/conftool/dbconfig/20240411-153003-arnaudb.json
• 15:28 marostegui@cumin1002: dbctl commit (dc=all): 'db2149 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P60432 and previous config saved to /var/cache/conftool/dbconfig/20240411-152825-root.json
• 15:24 hnowlan@deploy1002: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
• 15:24 hnowlan@deploy1002: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
• 15:24 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
• 15:23 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
• 15:20 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host moss-fe2002.codfw.wmnet with OS bookworm
• 15:18 mvernon@cumin1002: START - Cookbook sre.hosts.reimage for host moss-fe1002.eqiad.wmnet with OS bookworm
• 15:14 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on matomo1003.eqiad.wmnet with reason: host reimage
• 15:13 marostegui@cumin1002: dbctl commit (dc=all): 'db2149 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P60431 and previous config saved to /var/cache/conftool/dbconfig/20240411-151319-root.json
• 15:12 dancy@deploy1002: Finished scap: Backport for static.php: Handle mediawiki.org/ontology/ontology.owl (T171807 T359643) (duration: 17m 41s)
• 15:11 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on matomo1003.eqiad.wmnet with reason: host reimage
• 15:00 dancy@deploy1002: dancy: Continuing with sync
• 14:58 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 100%: Post upgrade', diff saved to https://phabricator.wikimedia.org/P60430 and previous config saved to /var/cache/conftool/dbconfig/20240411-145841-arnaudb.json
• 14:58 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host matomo1003.eqiad.wmnet with OS bookworm
• 14:58 marostegui@cumin1002: dbctl commit (dc=all): 'db2149 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P60429 and previous config saved to /var/cache/conftool/dbconfig/20240411-145813-root.json
• 14:57 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-drmrs and A:cp
• 14:57 dancy@deploy1002: dancy: Backport for static.php: Handle mediawiki.org/ontology/ontology.owl (T171807 T359643) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 14:56 arnaudb@cumin1002: dbctl commit (dc=all): 'db2115 (re)pooling @ 100%: Repool', diff saved to https://phabricator.wikimedia.org/P60428 and previous config saved to /var/cache/conftool/dbconfig/20240411-145658-arnaudb.json
• 14:54 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-codfw and not P{cp2042.codfw.wmnet} and A:cp
• 14:54 dancy@deploy1002: Started scap: Backport for static.php: Handle mediawiki.org/ontology/ontology.owl (T171807 T359643)
• 14:52 sukhe: sudo cumin "A:cp and A:esams" "run-puppet-agent --enable 'merging CR 1014571'"
• 14:52 dreamyjazz@deploy1002: Finished scap: Backport for Set wgMFFallbackEditor to visual for most VE wikis (T361134) (duration: 24m 11s)
• 14:47 hnowlan@deploy1002: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
• 14:47 hnowlan@deploy1002: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
• 14:47 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
• 14:45 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
• 14:44 jmm@cumin2002: END (PASS) - Cookbook sre.maps.roll-restart-reboot-master (exit_code=0) rolling restart_daemons on A:maps-master
• 14:44 arnaudb@cumin1002: dbctl commit (dc=all): 'db2103 (re)pooling @ 100%: reool', diff saved to https://phabricator.wikimedia.org/P60427 and previous config saved to /var/cache/conftool/dbconfig/20240411-144416-arnaudb.json
• 14:43 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 75%: Post upgrade', diff saved to https://phabricator.wikimedia.org/P60426 and previous config saved to /var/cache/conftool/dbconfig/20240411-144336-arnaudb.json
• 14:43 jmm@cumin2002: START - Cookbook sre.maps.roll-restart-reboot-master rolling restart_daemons on A:maps-master
• 14:43 arnaudb@cumin1002: dbctl commit (dc=all): 'db2109 (re)pooling @ 100%: repool', diff saved to https://phabricator.wikimedia.org/P60425 and previous config saved to /var/cache/conftool/dbconfig/20240411-144311-arnaudb.json
• 14:43 sukhe: sudo cumin "A:cp and A:esams" "disable-puppet 'merging CR 1014571'"
• 14:43 marostegui@cumin1002: dbctl commit (dc=all): 'db2149 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P60424 and previous config saved to /var/cache/conftool/dbconfig/20240411-144307-root.json
• 14:41 arnaudb@cumin1002: dbctl commit (dc=all): 'db2115 (re)pooling @ 75%: Repool', diff saved to https://phabricator.wikimedia.org/P60423 and previous config saved to /var/cache/conftool/dbconfig/20240411-144152-arnaudb.json
• 14:39 dreamyjazz@deploy1002: dreamyjazz and esanders: Continuing with sync
• 14:36 jmm@cumin2002: END (PASS) - Cookbook sre.maps.roll-restart-reboot (exit_code=0) rolling restart_daemons on A:maps-replica-eqiad
• 14:34 moritzm: installing distro-info-data updates from Bullseye point release
• 14:31 jmm@cumin2002: START - Cookbook sre.maps.roll-restart-reboot rolling restart_daemons on A:maps-replica-eqiad
• 14:30 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2149.codfw.wmnet with OS bookworm
• 14:30 dreamyjazz@deploy1002: dreamyjazz and esanders: Backport for Set wgMFFallbackEditor to visual for most VE wikis (T361134) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 14:29 arnaudb@cumin1002: dbctl commit (dc=all): 'db2103 (re)pooling @ 75%: reool', diff saved to https://phabricator.wikimedia.org/P60422 and previous config saved to /var/cache/conftool/dbconfig/20240411-142910-arnaudb.json
• 14:28 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 50%: Post upgrade', diff saved to https://phabricator.wikimedia.org/P60421 and previous config saved to /var/cache/conftool/dbconfig/20240411-142830-arnaudb.json
• 14:28 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp2042.codfw.wmnet,service=(cdn|ats-be)
• 14:28 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp3073.esams.wmnet,service=(cdn|ats-be)
• 14:28 arnaudb@cumin1002: dbctl commit (dc=all): 'db2109 (re)pooling @ 75%: repool', diff saved to https://phabricator.wikimedia.org/P60420 and previous config saved to /var/cache/conftool/dbconfig/20240411-142806-arnaudb.json
• 14:28 marostegui@cumin1002: dbctl commit (dc=all): 'db2149 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P60419 and previous config saved to /var/cache/conftool/dbconfig/20240411-142801-root.json
• 14:27 dreamyjazz@deploy1002: Started scap: Backport for Set wgMFFallbackEditor to visual for most VE wikis (T361134)
• 14:27 Dreamy_Jazz: Extending UTC Afternoon backport window
• 14:26 arnaudb@cumin1002: dbctl commit (dc=all): 'db2115 (re)pooling @ 50%: Repool', diff saved to https://phabricator.wikimedia.org/P60418 and previous config saved to /var/cache/conftool/dbconfig/20240411-142645-arnaudb.json
• 14:26 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp2042.codfw.wmnet with OS bullseye
• 14:24 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3073.esams.wmnet with OS bullseye
• 14:19 jmm@cumin2002: END (PASS) - Cookbook sre.maps.roll-restart-reboot (exit_code=0) rolling restart_daemons on A:maps-replica-codfw
• 14:18 elukey: drain and restart cassandra-b on aqs2007 - didn't pick up the new truststore during the past roll restart - T352647
• 14:15 jmm@cumin2002: START - Cookbook sre.maps.roll-restart-reboot rolling restart_daemons on A:maps-replica-codfw
• 14:14 arnaudb@cumin1002: dbctl commit (dc=all): 'db2103 (re)pooling @ 50%: reool', diff saved to https://phabricator.wikimedia.org/P60417 and previous config saved to /var/cache/conftool/dbconfig/20240411-141404-arnaudb.json
• 14:13 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 25%: Post upgrade', diff saved to https://phabricator.wikimedia.org/P60416 and previous config saved to /var/cache/conftool/dbconfig/20240411-141324-arnaudb.json
• 14:13 arnaudb@cumin1002: dbctl commit (dc=all): 'db2109 (re)pooling @ 50%: repool', diff saved to https://phabricator.wikimedia.org/P60415 and previous config saved to /var/cache/conftool/dbconfig/20240411-141300-arnaudb.json
• 14:12 hnowlan@deploy1002: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
• 14:11 arnaudb@cumin1002: dbctl commit (dc=all): 'db2115 (re)pooling @ 25%: Repool', diff saved to https://phabricator.wikimedia.org/P60414 and previous config saved to /var/cache/conftool/dbconfig/20240411-141139-arnaudb.json
• 14:11 hnowlan@deploy1002: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
• 14:10 elukey: move cassandra instances on aqs1010 to PKI TLS certs - T352647
• 14:10 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
• 14:10 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2149.codfw.wmnet with reason: host reimage
• 14:09 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
• 14:09 Dreamy_Jazz: Afternoon UTC backport window finished
• 14:09 moritzm: installing NSS security updates
• 14:08 dreamyjazz@deploy1002: Finished scap: Backport for Ignore missing title/page in CheckUserLookupUtils::getManualLogEntryFromRow (T362284) (duration: 17m 42s)
• 14:06 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2149.codfw.wmnet with reason: host reimage
• 14:06 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp2042.codfw.wmnet with reason: host reimage
• 14:06 btullis@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host matomo1003.eqiad.wmnet with OS bookworm
• 14:03 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp2042.codfw.wmnet with reason: host reimage
• 14:01 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3073.esams.wmnet with reason: host reimage
• 13:59 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on aqs1010.eqiad.wmnet with reason: Upgrade to PKI
• 13:59 elukey@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on aqs1010.eqiad.wmnet with reason: Upgrade to PKI
• 13:59 arnaudb@cumin1002: dbctl commit (dc=all): 'db2103 (re)pooling @ 25%: reool', diff saved to https://phabricator.wikimedia.org/P60413 and previous config saved to /var/cache/conftool/dbconfig/20240411-135858-arnaudb.json
• 13:58 marostegui@cumin1002: dbctl commit (dc=all): 'db2177 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P60412 and previous config saved to /var/cache/conftool/dbconfig/20240411-135846-root.json
• 13:58 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 20%: Post upgrade', diff saved to https://phabricator.wikimedia.org/P60411 and previous config saved to /var/cache/conftool/dbconfig/20240411-135819-arnaudb.json
• 13:58 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3073.esams.wmnet with reason: host reimage
• 13:57 arnaudb@cumin1002: dbctl commit (dc=all): 'db2109 (re)pooling @ 25%: repool', diff saved to https://phabricator.wikimedia.org/P60410 and previous config saved to /var/cache/conftool/dbconfig/20240411-135754-arnaudb.json
• 13:57 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-codfw and not P{cp2042.codfw.wmnet} and A:cp
• 13:56 arnaudb@cumin1002: dbctl commit (dc=all): 'db2115 (re)pooling @ 10%: Repool', diff saved to https://phabricator.wikimedia.org/P60409 and previous config saved to /var/cache/conftool/dbconfig/20240411-135634-arnaudb.json
• 13:55 dreamyjazz@deploy1002: dreamyjazz: Continuing with sync
• 13:55 dreamyjazz@deploy1002: dreamyjazz: Backport for Ignore missing title/page in CheckUserLookupUtils::getManualLogEntryFromRow (T362284) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 13:54 ayounsi@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts testvm2008.wikimedia.org
• 13:54 ayounsi@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 13:54 ayounsi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: testvm2008.wikimedia.org decommissioned, removing all IPs except the asset tag one - ayounsi@cumin1002"
• 13:53 ayounsi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: testvm2008.wikimedia.org decommissioned, removing all IPs except the asset tag one - ayounsi@cumin1002"
• 13:51 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db2149.codfw.wmnet with OS bookworm
• 13:50 dreamyjazz@deploy1002: Started scap: Backport for Ignore missing title/page in CheckUserLookupUtils::getManualLogEntryFromRow (T362284)
• 13:49 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2149', diff saved to https://phabricator.wikimedia.org/P60408 and previous config saved to /var/cache/conftool/dbconfig/20240411-134932-root.json
• 13:49 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host matomo1003.eqiad.wmnet with OS bookworm
• 13:46 btullis@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host matomo1003.eqiad.wmnet with OS bookworm
• 13:46 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp2042.codfw.wmnet with OS bullseye
• 13:45 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp2042.codfw.wmnet,service=(cdn|ats-be)
• 13:43 marostegui@cumin1002: dbctl commit (dc=all): 'db2177 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P60407 and previous config saved to /var/cache/conftool/dbconfig/20240411-134341-root.json
• 13:43 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 10%: Post upgrade', diff saved to https://phabricator.wikimedia.org/P60406 and previous config saved to /var/cache/conftool/dbconfig/20240411-134312-arnaudb.json
• 13:41 jmm@cumin2002: END (PASS) - Cookbook sre.maps.roll-restart-reboot (exit_code=0) rolling restart_daemons on A:maps-replica-eqiad
• 13:36 jmm@cumin2002: START - Cookbook sre.maps.roll-restart-reboot rolling restart_daemons on A:maps-replica-eqiad
• 13:34 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp3073.esams.wmnet with OS bullseye
• 13:32 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp3073.esams.wmnet,service=(cdn|ats-be)
• 13:32 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on db2160.codfw.wmnet with reason: reboot multiinstance replica
• 13:32 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 0:30:00 on db2160.codfw.wmnet with reason: reboot multiinstance replica
• 13:32 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host matomo1003.eqiad.wmnet with OS bookworm
• 13:31 btullis@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host matomo1003.eqiad.wmnet with OS bookworm
• 13:30 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2133.codfw.wmnet
• 13:28 marostegui@cumin1002: dbctl commit (dc=all): 'db2177 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P60405 and previous config saved to /var/cache/conftool/dbconfig/20240411-132834-root.json
• 13:28 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 5%: Post upgrade', diff saved to https://phabricator.wikimedia.org/P60404 and previous config saved to /var/cache/conftool/dbconfig/20240411-132807-arnaudb.json
• 13:27 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-eqsin and not P{cp[5030,5032].eqsin.wmnet} and A:cp
• 13:26 arnaudb@cumin1002: START - Cookbook sre.mysql.upgrade for db2133.codfw.wmnet
• 13:26 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on db[2133,2160].codfw.wmnet with reason: reboot
• 13:25 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 0:30:00 on db[2133,2160].codfw.wmnet with reason: reboot
• 13:23 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2135.codfw.wmnet
• 13:18 arnaudb@cumin1002: START - Cookbook sre.mysql.upgrade for db2135.codfw.wmnet
• 13:17 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on db[2135,2160].codfw.wmnet with reason: reboot
• 13:17 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 0:30:00 on db[2135,2160].codfw.wmnet with reason: reboot
• 13:16 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2134.codfw.wmnet
• 13:13 marostegui@cumin1002: dbctl commit (dc=all): 'db2177 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P60403 and previous config saved to /var/cache/conftool/dbconfig/20240411-131327-root.json
• 13:13 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 4%: Post upgrade', diff saved to https://phabricator.wikimedia.org/P60402 and previous config saved to /var/cache/conftool/dbconfig/20240411-131301-arnaudb.json
• 13:12 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host matomo1003.eqiad.wmnet with OS bookworm
• 13:12 arnaudb@cumin1002: START - Cookbook sre.mysql.upgrade for db2134.codfw.wmnet
• 13:12 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on db[2134,2160].codfw.wmnet with reason: reboot
• 13:11 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 0:30:00 on db[2134,2160].codfw.wmnet with reason: reboot
• 13:00 btullis@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host matomo1003.eqiad.wmnet with OS bookworm
• 12:58 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2132.codfw.wmnet
• 12:58 marostegui@cumin1002: dbctl commit (dc=all): 'db2177 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P60401 and previous config saved to /var/cache/conftool/dbconfig/20240411-125821-root.json
• 12:57 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 2%: Post upgrade', diff saved to https://phabricator.wikimedia.org/P60400 and previous config saved to /var/cache/conftool/dbconfig/20240411-125755-arnaudb.json
• 12:54 akosiaris: lower weight of mw1437 back to 10 from the 30 I had upped it to yesterday. The backlog of videoscaling is apparently now served and CPU usage has reached "normal" levels
• 12:54 arnaudb@cumin1002: START - Cookbook sre.mysql.upgrade for db2132.codfw.wmnet
• 12:54 jayme@deploy1002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 12:53 akosiaris@cumin1002: conftool action : set/weight=10; selector: name=mw1437.*.wmnet,dc=eqiad
• 12:53 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db[2132,2160].codfw.wmnet with reason: reboot
• 12:53 jayme@deploy1002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 12:53 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db[2132,2160].codfw.wmnet with reason: reboot
• 12:52 jayme@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 12:52 jayme@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'.
• 12:51 jayme@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
• 12:50 jayme@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
• 12:49 jayme@deploy1002: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
• 12:49 jayme@deploy1002: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
• 12:45 ayounsi@cumin1002: START - Cookbook sre.dns.netbox
• 12:24 btullis@deploy1002: helmfile [eqiad] DONE helmfile.d/services/editor-analytics: apply
• 12:24 btullis@deploy1002: helmfile [eqiad] START helmfile.d/services/editor-analytics: apply
• 12:23 btullis@deploy1002: helmfile [codfw] DONE helmfile.d/services/editor-analytics: apply
• 12:22 btullis@deploy1002: helmfile [codfw] START helmfile.d/services/editor-analytics: apply
• 12:21 ayounsi@cumin1002: START - Cookbook sre.hosts.reimage for host testvm2008.wikimedia.org with OS bookworm
• 12:21 ayounsi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM testvm2008.wikimedia.org - ayounsi@cumin1002"
• 12:20 btullis@deploy1002: helmfile [staging] DONE helmfile.d/services/editor-analytics: apply
• 12:20 ayounsi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM testvm2008.wikimedia.org - ayounsi@cumin1002"
• 12:20 ayounsi@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) testvm2008.wikimedia.org on all recursors
• 12:20 btullis@deploy1002: helmfile [staging] START helmfile.d/services/editor-analytics: apply
• 12:19 ayounsi@cumin1002: START - Cookbook sre.dns.wipe-cache testvm2008.wikimedia.org on all recursors
• 12:19 ayounsi@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:19 ayounsi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2008.wikimedia.org - ayounsi@cumin1002"
• 12:18 ayounsi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2008.wikimedia.org - ayounsi@cumin1002"
• 12:16 hnowlan@deploy1002: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
• 12:16 ayounsi@cumin1002: START - Cookbook sre.dns.netbox
• 12:16 ayounsi@cumin1002: START - Cookbook sre.ganeti.makevm for new host testvm2008.wikimedia.org
• 12:16 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host matomo1003.eqiad.wmnet with OS bookworm
• 12:16 ayounsi@cumin1002: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host testvm2008.wikimedia.org
• 12:16 ayounsi@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=97)
• 12:16 ayounsi@cumin1002: END (ERROR) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=97) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2008.wikimedia.org - ayounsi@cumin1002"
• 12:16 ayounsi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2008.wikimedia.org - ayounsi@cumin1002"
• 12:15 hnowlan@deploy1002: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
• 12:15 moritzm: installing gnutls28 security updates
• 12:14 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
• 12:13 ayounsi@cumin1002: START - Cookbook sre.dns.netbox
• 12:13 ayounsi@cumin1002: START - Cookbook sre.ganeti.makevm for new host testvm2008.wikimedia.org
• 12:13 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
• 12:13 ayounsi@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts testvm2008.wikimedia.org
• 12:13 ayounsi@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:13 ayounsi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: testvm2008.wikimedia.org decommissioned, removing all IPs except the asset tag one - ayounsi@cumin1002"
• 12:13 btullis@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host matomo1003.eqiad.wmnet with OS bullseye
• 12:12 ayounsi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: testvm2008.wikimedia.org decommissioned, removing all IPs except the asset tag one - ayounsi@cumin1002"
• 12:10 ayounsi@cumin1002: START - Cookbook sre.dns.netbox
• 12:06 ayounsi@cumin1002: START - Cookbook sre.hosts.decommission for hosts testvm2008.wikimedia.org
• 12:06 ayounsi@cumin1002: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=97) for new host testvm2008.wikimedia.org
• 12:06 ayounsi@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host testvm2008.wikimedia.org with OS bookworm
• 12:02 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
• 12:01 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
• 12:01 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
• 11:59 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
• 11:59 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
• 11:58 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
• 11:58 ayounsi@cumin1002: START - Cookbook sre.hosts.reimage for host testvm2008.wikimedia.org with OS bookworm
• 11:57 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-web: apply
• 11:57 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/services/mw-web: apply
• 11:52 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2177.codfw.wmnet with OS bookworm
• 11:50 ayounsi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM testvm2008.wikimedia.org - ayounsi@cumin1002"
• 11:50 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on matomo1003.eqiad.wmnet with reason: host reimage
• 11:49 ayounsi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM testvm2008.wikimedia.org - ayounsi@cumin1002"
• 11:49 ayounsi@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) testvm2008.wikimedia.org on all recursors
• 11:49 ayounsi@cumin1002: START - Cookbook sre.dns.wipe-cache testvm2008.wikimedia.org on all recursors
• 11:49 ayounsi@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 11:49 ayounsi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2008.wikimedia.org - ayounsi@cumin1002"
• 11:47 ayounsi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2008.wikimedia.org - ayounsi@cumin1002"
• 11:47 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on matomo1003.eqiad.wmnet with reason: host reimage
• 11:45 ayounsi@cumin1002: START - Cookbook sre.dns.netbox
• 11:45 ayounsi@cumin1002: START - Cookbook sre.ganeti.makevm for new host testvm2008.wikimedia.org
• 11:33 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host matomo1003.eqiad.wmnet with OS bullseye
• 11:31 moritzm: installing postgresql-15 security updates
• 11:31 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2177.codfw.wmnet with reason: host reimage
• 11:27 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2177.codfw.wmnet with reason: host reimage
• 11:24 effie: upload prometheus-memcached-exporter 0.14.2-1~wmf1 to bookworm-wikimedia main - T350807
• 11:22 effie: upload memkeys 20181031-2-s1 to bookworm-wikimedia main - T362160
• 11:22 effie: upload memkeys 20181031-2-s1 to bookworm-wikimedia main
• 11:10 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db2177.codfw.wmnet with OS bookworm
• 11:09 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2177', diff saved to https://phabricator.wikimedia.org/P60394 and previous config saved to /var/cache/conftool/dbconfig/20240411-110938-root.json
• 10:53 cgoubert@cumin1002: conftool action : set/weight=10:pooled=yes; selector: name=(mw2412.codfw.wmnet|mw2413.codfw.wmnet|mw2414.codfw.wmnet|mw2415.codfw.wmnet|mw2416.codfw.wmnet|mw2417.codfw.wmnet|mw2418.codfw.wmnet),cluster=kubernetes,service=kubesvc
• 10:52 claime: Pooling and uncordoning mw2412.codfw.wmnet,mw2413.codfw.wmnet,mw2414.codfw.wmnet,mw2415.codfw.wmnet,mw2416.codfw.wmnet,mw2417.codfw.wmnet,mw2418.codfw.wmnet - T351074
• 10:43 moritzm: installing modsecurity-apache security updates
• 10:37 claime: Running homer 'cr*codfw*' commit 'T351074'
• 10:36 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2413.codfw.wmnet with OS bullseye
• 10:31 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2418.codfw.wmnet with OS bullseye
• 10:30 moritzm: installing xerces-c security updates
• 10:30 marostegui@cumin1002: dbctl commit (dc=all): 'db1198 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P60393 and previous config saved to /var/cache/conftool/dbconfig/20240411-103005-root.json
• 10:28 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2412.codfw.wmnet with OS bullseye
• 10:25 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2415.codfw.wmnet with OS bullseye
• 10:22 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2416.codfw.wmnet with OS bullseye
• 10:21 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1247 (T356166)', diff saved to https://phabricator.wikimedia.org/P60392 and previous config saved to /var/cache/conftool/dbconfig/20240411-102153-marostegui.json
• 10:21 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1247.eqiad.wmnet with reason: Maintenance
• 10:21 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1247.eqiad.wmnet with reason: Maintenance
• 10:20 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 100%: post schema update', diff saved to https://phabricator.wikimedia.org/P60391 and previous config saved to /var/cache/conftool/dbconfig/20240411-102031-arnaudb.json
• 10:19 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2417.codfw.wmnet with OS bullseye
• 10:17 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2413.codfw.wmnet with reason: host reimage
• 10:15 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2414.codfw.wmnet with OS bullseye
• 10:15 marostegui@cumin1002: dbctl commit (dc=all): 'db1198 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P60390 and previous config saved to /var/cache/conftool/dbconfig/20240411-101500-root.json
• 10:13 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2418.codfw.wmnet with reason: host reimage
• 10:09 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2412.codfw.wmnet with reason: host reimage
• 10:06 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2415.codfw.wmnet with reason: host reimage
• 10:05 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 75%: post schema update', diff saved to https://phabricator.wikimedia.org/P60389 and previous config saved to /var/cache/conftool/dbconfig/20240411-100525-arnaudb.json
• 10:03 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2416.codfw.wmnet with reason: host reimage
• 10:00 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2417.codfw.wmnet with reason: host reimage
• 09:59 marostegui@cumin1002: dbctl commit (dc=all): 'db1198 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P60388 and previous config saved to /var/cache/conftool/dbconfig/20240411-095954-root.json
• 09:57 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw2418.codfw.wmnet with reason: host reimage
• 09:57 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2414.codfw.wmnet with reason: host reimage
• 09:57 ayounsi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "testvm2007 - ayounsi@cumin1002"
• 09:57 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw2417.codfw.wmnet with reason: host reimage
• 09:56 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw2416.codfw.wmnet with reason: host reimage
• 09:56 ayounsi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "testvm2007 - ayounsi@cumin1002"
• 09:56 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw2415.codfw.wmnet with reason: host reimage
• 09:55 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw2413.codfw.wmnet with reason: host reimage
• 09:55 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw2412.codfw.wmnet with reason: host reimage
• 09:55 ayounsi@cumin1002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host testvm2007.codfw.wmnet
• 09:55 ayounsi@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host testvm2007.codfw.wmnet with OS bookworm
• 09:54 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw2414.codfw.wmnet with reason: host reimage
• 09:50 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 50%: post schema update', diff saved to https://phabricator.wikimedia.org/P60387 and previous config saved to /var/cache/conftool/dbconfig/20240411-095019-arnaudb.json
• 09:44 marostegui@cumin1002: dbctl commit (dc=all): 'db1198 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P60386 and previous config saved to /var/cache/conftool/dbconfig/20240411-094448-root.json
• 09:40 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw2418.codfw.wmnet with OS bullseye
• 09:40 ayounsi@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm2007.codfw.wmnet with reason: host reimage
• 09:40 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw2417.codfw.wmnet with OS bullseye
• 09:39 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw2416.codfw.wmnet with OS bullseye
• 09:39 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw2415.codfw.wmnet with OS bullseye
• 09:38 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw2414.codfw.wmnet with OS bullseye
• 09:38 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw2413.codfw.wmnet with OS bullseye
• 09:38 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw2412.codfw.wmnet with OS bullseye
• 09:38 ayounsi@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on testvm2007.codfw.wmnet with reason: host reimage
• 09:37 fabfur@cumin1002: conftool action : set/pooled=yes; selector: name=cp3072.esams.wmnet
• 09:35 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 25%: post schema update', diff saved to https://phabricator.wikimedia.org/P60384 and previous config saved to /var/cache/conftool/dbconfig/20240411-093513-arnaudb.json
• 09:32 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3072.esams.wmnet with OS bullseye
• 09:31 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2129.codfw.wmnet with reason: Maintenance
• 09:31 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2129.codfw.wmnet with reason: Maintenance
• 09:29 marostegui@cumin1002: dbctl commit (dc=all): 'db1198 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P60383 and previous config saved to /var/cache/conftool/dbconfig/20240411-092942-root.json
• 09:27 arnaudb@cumin1002: dbctl restore of MediaWiki config (dc=all) from /var/cache/conftool/dbconfig/20240411-092622-arnaudb.json
• 09:26 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1165 (T360332)', diff saved to https://phabricator.wikimedia.org/P60382 and previous config saved to /var/cache/conftool/dbconfig/20240411-092622-arnaudb.json
• 09:26 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
• 09:26 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
• 09:26 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1165.eqiad.wmnet with reason: Maintenance
• 09:26 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1165.eqiad.wmnet with reason: Maintenance
• 09:25 ayounsi@cumin1002: START - Cookbook sre.hosts.reimage for host testvm2007.codfw.wmnet with OS bookworm
• 09:25 ayounsi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM testvm2007.codfw.wmnet - ayounsi@cumin1002"
• 09:25 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 depool', diff saved to https://phabricator.wikimedia.org/P60381 and previous config saved to /var/cache/conftool/dbconfig/20240411-092501-arnaudb.json
• 09:24 ayounsi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM testvm2007.codfw.wmnet - ayounsi@cumin1002"
• 09:24 ayounsi@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) testvm2007.codfw.wmnet on all recursors
• 09:24 ayounsi@cumin1002: START - Cookbook sre.dns.wipe-cache testvm2007.codfw.wmnet on all recursors
• 09:24 ayounsi@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 09:24 ayounsi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2007.codfw.wmnet - ayounsi@cumin1002"
• 09:23 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 weight bump T362302', diff saved to https://phabricator.wikimedia.org/P60380 and previous config saved to /var/cache/conftool/dbconfig/20240411-092318-arnaudb.json
• 09:20 ayounsi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2007.codfw.wmnet - ayounsi@cumin1002"
• 09:20 arnaudb@cumin1002: dbctl commit (dc=all): 'Promote db2114 to s6 primary T362302', diff saved to https://phabricator.wikimedia.org/P60379 and previous config saved to /var/cache/conftool/dbconfig/20240411-092012-arnaudb.json
• 09:19 arnaudb: Starting s6 codfw failover from db2129 to db2114 - T362302
• 09:16 ayounsi@cumin1002: START - Cookbook sre.dns.netbox
• 09:16 ayounsi@cumin1002: START - Cookbook sre.ganeti.makevm for new host testvm2007.codfw.wmnet
• 09:13 jelto@cumin1002: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab Replica to new version
• 09:12 marostegui@cumin1002: dbctl commit (dc=all): 'db1198 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P60378 and previous config saved to /var/cache/conftool/dbconfig/20240411-091255-root.json
• 09:12 jelto@cumin1002: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab Replica to new version
• 09:06 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3072.esams.wmnet with reason: host reimage
• 09:03 fabfur@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3072.esams.wmnet with reason: host reimage
• 08:59 arnaudb@cumin1002: dbctl commit (dc=all): 'Set db2114 with weight 0 T362302', diff saved to https://phabricator.wikimedia.org/P60377 and previous config saved to /var/cache/conftool/dbconfig/20240411-085926-arnaudb.json
• 08:59 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 27 hosts with reason: Primary switchover s6 T362302
• 08:58 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on 27 hosts with reason: Primary switchover s6 T362302
• 08:58 ayounsi@cumin1002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host testvm2006.codfw.wmnet
• 08:58 ayounsi@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host testvm2006.codfw.wmnet with OS bookworm
• 08:57 marostegui@cumin1002: dbctl commit (dc=all): 'db1198 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P60376 and previous config saved to /var/cache/conftool/dbconfig/20240411-085749-root.json
• 08:55 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1198.eqiad.wmnet with OS bookworm
• 08:50 jelto@cumin1002: END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab Replica to new version
• 08:45 ayounsi@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm2006.codfw.wmnet with reason: host reimage
• 08:45 jelto@cumin1002: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab Replica to new version
• 08:45 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on matomo1003.eqiad.wmnet with reason: Adding disk
• 08:45 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on matomo1003.eqiad.wmnet with reason: Adding disk
• 08:42 jelto@cumin1002: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Upgrade GitLab Replica to new version
• 08:42 ayounsi@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on testvm2006.codfw.wmnet with reason: host reimage
• 08:40 fabfur@cumin1002: START - Cookbook sre.hosts.reimage for host cp3072.esams.wmnet with OS bullseye
• 08:40 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1198.eqiad.wmnet with reason: host reimage
• 08:37 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1198.eqiad.wmnet with reason: host reimage
• 08:36 jelto@cumin1002: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Upgrade GitLab Replica to new version
• 08:36 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db1198.eqiad.wmnet with OS bookworm
• 08:31 ayounsi@cumin1002: START - Cookbook sre.hosts.reimage for host testvm2006.codfw.wmnet with OS bookworm
• 08:29 ayounsi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM testvm2006.codfw.wmnet - ayounsi@cumin1002"
• 08:29 ayounsi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM testvm2006.codfw.wmnet - ayounsi@cumin1002"
• 08:28 ayounsi@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) testvm2006.codfw.wmnet on all recursors
• 08:28 ayounsi@cumin1002: START - Cookbook sre.dns.wipe-cache testvm2006.codfw.wmnet on all recursors
• 08:28 ayounsi@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 08:28 ayounsi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2006.codfw.wmnet - ayounsi@cumin1002"
• 08:27 marostegui@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host db1198.eqiad.wmnet with OS bookworm
• 08:27 ayounsi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2006.codfw.wmnet - ayounsi@cumin1002"
• 08:26 fabfur@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp3072.esams.wmnet with OS bullseye
• 08:25 ayounsi@cumin1002: START - Cookbook sre.dns.netbox
• 08:25 ayounsi@cumin1002: START - Cookbook sre.ganeti.makevm for new host testvm2006.codfw.wmnet
• 08:22 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1198.eqiad.wmnet with reason: host reimage
• 08:20 hashar: MediaWiki train is blocked
• 08:19 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1198.eqiad.wmnet with reason: host reimage
• 08:13 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3072.esams.wmnet with reason: host reimage
• 08:10 fabfur@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3072.esams.wmnet with reason: host reimage
• 08:06 slyngshede@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp2002.wikimedia.org
• 08:06 slyngshede@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 08:06 slyngshede@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: idp2002.wikimedia.org decommissioned, removing all IPs except the asset tag one - slyngshede@cumin1002"
• 08:06 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db1198.eqiad.wmnet with OS bookworm
• 08:05 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1198', diff saved to https://phabricator.wikimedia.org/P60374 and previous config saved to /var/cache/conftool/dbconfig/20240411-080502-root.json
• 08:03 slyngshede@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: idp2002.wikimedia.org decommissioned, removing all IPs except the asset tag one - slyngshede@cumin1002"
• 08:01 slyngshede@cumin1002: START - Cookbook sre.dns.netbox
• 07:56 slyngshede@cumin1002: START - Cookbook sre.hosts.decommission for hosts idp2002.wikimedia.org
• 07:47 fabfur@cumin1002: START - Cookbook sre.hosts.reimage for host cp3072.esams.wmnet with OS bullseye
• 07:44 fabfur@cumin1002: conftool action : set/pooled=no; selector: name=cp3072.esams.wmnet
• 07:39 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 07:39 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 07:25 marostegui@cumin1002: dbctl commit (dc=all): 'db1189 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P60373 and previous config saved to /var/cache/conftool/dbconfig/20240411-072503-root.json
• 07:10 slyngshede@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp1002.wikimedia.org
• 07:10 slyngshede@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 07:10 slyngshede@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: idp1002.wikimedia.org decommissioned, removing all IPs except the asset tag one - slyngshede@cumin1002"
• 07:09 marostegui@cumin1002: dbctl commit (dc=all): 'db1189 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P60372 and previous config saved to /var/cache/conftool/dbconfig/20240411-070958-root.json
• 07:08 slyngshede@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: idp1002.wikimedia.org decommissioned, removing all IPs except the asset tag one - slyngshede@cumin1002"
• 07:05 slyngshede@cumin1002: START - Cookbook sre.dns.netbox
• 07:00 slyngshede@cumin1002: START - Cookbook sre.hosts.decommission for hosts idp1002.wikimedia.org
• 06:57 slyngshede@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts idp1002.wikimedia.org
• 06:56 slyngshede@cumin1002: START - Cookbook sre.hosts.decommission for hosts idp1002.wikimedia.org
• 06:54 marostegui@cumin1002: dbctl commit (dc=all): 'db1189 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P60371 and previous config saved to /var/cache/conftool/dbconfig/20240411-065452-root.json
• 06:39 marostegui@cumin1002: dbctl commit (dc=all): 'db1189 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P60370 and previous config saved to /var/cache/conftool/dbconfig/20240411-063946-root.json
• 06:27 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2216 (T360332)', diff saved to https://phabricator.wikimedia.org/P60369 and previous config saved to /var/cache/conftool/dbconfig/20240411-062728-arnaudb.json
• 06:24 marostegui@cumin1002: dbctl commit (dc=all): 'db1189 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P60368 and previous config saved to /var/cache/conftool/dbconfig/20240411-062440-root.json
• 06:12 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2216', diff saved to https://phabricator.wikimedia.org/P60367 and previous config saved to /var/cache/conftool/dbconfig/20240411-061220-arnaudb.json
• 06:09 marostegui@cumin1002: dbctl commit (dc=all): 'db1189 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P60366 and previous config saved to /var/cache/conftool/dbconfig/20240411-060934-root.json
• 05:57 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2216', diff saved to https://phabricator.wikimedia.org/P60365 and previous config saved to /var/cache/conftool/dbconfig/20240411-055712-arnaudb.json
• 05:54 marostegui@cumin1002: dbctl commit (dc=all): 'db1189 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P60364 and previous config saved to /var/cache/conftool/dbconfig/20240411-055428-root.json
• 05:52 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1189.eqiad.wmnet with OS bookworm
• 05:42 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2216 (T360332)', diff saved to https://phabricator.wikimedia.org/P60363 and previous config saved to /var/cache/conftool/dbconfig/20240411-054205-arnaudb.json
• 05:39 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2216 (T360332)', diff saved to https://phabricator.wikimedia.org/P60362 and previous config saved to /var/cache/conftool/dbconfig/20240411-053903-arnaudb.json
• 05:38 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2216.codfw.wmnet with reason: Maintenance
• 05:38 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2216.codfw.wmnet with reason: Maintenance
• 05:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2212 (T360332)', diff saved to https://phabricator.wikimedia.org/P60361 and previous config saved to /var/cache/conftool/dbconfig/20240411-053840-arnaudb.json
• 05:31 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1189.eqiad.wmnet with reason: host reimage
• 05:27 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1189.eqiad.wmnet with reason: host reimage
• 05:23 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2212', diff saved to https://phabricator.wikimedia.org/P60360 and previous config saved to /var/cache/conftool/dbconfig/20240411-052333-arnaudb.json
• 05:15 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db1189.eqiad.wmnet with OS bookworm
• 05:13 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1189', diff saved to https://phabricator.wikimedia.org/P60359 and previous config saved to /var/cache/conftool/dbconfig/20240411-051341-root.json
• 05:08 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2212', diff saved to https://phabricator.wikimedia.org/P60358 and previous config saved to /var/cache/conftool/dbconfig/20240411-050825-arnaudb.json
• 04:53 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2212 (T360332)', diff saved to https://phabricator.wikimedia.org/P60357 and previous config saved to /var/cache/conftool/dbconfig/20240411-045317-arnaudb.json
• 04:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2212 (T360332)', diff saved to https://phabricator.wikimedia.org/P60356 and previous config saved to /var/cache/conftool/dbconfig/20240411-045024-arnaudb.json
• 04:50 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2212.codfw.wmnet with reason: Maintenance
• 04:50 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2212.codfw.wmnet with reason: Maintenance
• 04:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2188 (T360332)', diff saved to https://phabricator.wikimedia.org/P60355 and previous config saved to /var/cache/conftool/dbconfig/20240411-045011-arnaudb.json
• 04:35 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2188', diff saved to https://phabricator.wikimedia.org/P60354 and previous config saved to /var/cache/conftool/dbconfig/20240411-043502-arnaudb.json
• 04:19 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2188', diff saved to https://phabricator.wikimedia.org/P60353 and previous config saved to /var/cache/conftool/dbconfig/20240411-041954-arnaudb.json
• 04:04 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2188 (T360332)', diff saved to https://phabricator.wikimedia.org/P60352 and previous config saved to /var/cache/conftool/dbconfig/20240411-040447-arnaudb.json
• 04:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2188 (T360332)', diff saved to https://phabricator.wikimedia.org/P60351 and previous config saved to /var/cache/conftool/dbconfig/20240411-040147-arnaudb.json
• 04:01 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2188.codfw.wmnet with reason: Maintenance
• 04:01 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2188.codfw.wmnet with reason: Maintenance
• 04:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2176 (T360332)', diff saved to https://phabricator.wikimedia.org/P60350 and previous config saved to /var/cache/conftool/dbconfig/20240411-040124-arnaudb.json
• 03:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P60349 and previous config saved to /var/cache/conftool/dbconfig/20240411-034617-arnaudb.json
• 03:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P60348 and previous config saved to /var/cache/conftool/dbconfig/20240411-033109-arnaudb.json
• 03:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2176 (T360332)', diff saved to https://phabricator.wikimedia.org/P60347 and previous config saved to /var/cache/conftool/dbconfig/20240411-031602-arnaudb.json
• 03:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2176 (T360332)', diff saved to https://phabricator.wikimedia.org/P60346 and previous config saved to /var/cache/conftool/dbconfig/20240411-031310-arnaudb.json
• 03:13 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2176.codfw.wmnet with reason: Maintenance
• 03:12 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2176.codfw.wmnet with reason: Maintenance
• 03:12 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2174 (T360332)', diff saved to https://phabricator.wikimedia.org/P60345 and previous config saved to /var/cache/conftool/dbconfig/20240411-031247-arnaudb.json
• 02:57 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P60344 and previous config saved to /var/cache/conftool/dbconfig/20240411-025740-arnaudb.json
• 02:42 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P60343 and previous config saved to /var/cache/conftool/dbconfig/20240411-024232-arnaudb.json
• 02:31 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1245.eqiad.wmnet with reason: Maintenance
• 02:31 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1245.eqiad.wmnet with reason: Maintenance
• 02:31 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1244 (T356166)', diff saved to https://phabricator.wikimedia.org/P60342 and previous config saved to /var/cache/conftool/dbconfig/20240411-023125-marostegui.json
• 02:30 cstone: civicrm upgraded from a382a7b0 to c2569254
• 02:27 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2174 (T360332)', diff saved to https://phabricator.wikimedia.org/P60341 and previous config saved to /var/cache/conftool/dbconfig/20240411-022725-arnaudb.json
• 02:24 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2174 (T360332)', diff saved to https://phabricator.wikimedia.org/P60340 and previous config saved to /var/cache/conftool/dbconfig/20240411-022433-arnaudb.json
• 02:24 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2174.codfw.wmnet with reason: Maintenance
• 02:24 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2174.codfw.wmnet with reason: Maintenance
• 02:24 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2173 (T360332)', diff saved to https://phabricator.wikimedia.org/P60339 and previous config saved to /var/cache/conftool/dbconfig/20240411-022410-arnaudb.json
• 02:16 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P60338 and previous config saved to /var/cache/conftool/dbconfig/20240411-021617-marostegui.json
• 02:09 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P60337 and previous config saved to /var/cache/conftool/dbconfig/20240411-020903-arnaudb.json
• 02:01 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P60336 and previous config saved to /var/cache/conftool/dbconfig/20240411-020110-marostegui.json
• 01:53 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P60335 and previous config saved to /var/cache/conftool/dbconfig/20240411-015355-arnaudb.json
• 01:46 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1244 (T356166)', diff saved to https://phabricator.wikimedia.org/P60334 and previous config saved to /var/cache/conftool/dbconfig/20240411-014602-marostegui.json
• 01:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2173 (T360332)', diff saved to https://phabricator.wikimedia.org/P60333 and previous config saved to /var/cache/conftool/dbconfig/20240411-013848-arnaudb.json
• 01:36 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2173 (T360332)', diff saved to https://phabricator.wikimedia.org/P60332 and previous config saved to /var/cache/conftool/dbconfig/20240411-013657-arnaudb.json
• 01:36 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance
• 01:36 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance
• 01:36 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2173.codfw.wmnet with reason: Maintenance
• 01:36 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2173.codfw.wmnet with reason: Maintenance
• 01:36 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2170 (T360332)', diff saved to https://phabricator.wikimedia.org/P60331 and previous config saved to /var/cache/conftool/dbconfig/20240411-013618-arnaudb.json
• 01:21 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2170', diff saved to https://phabricator.wikimedia.org/P60330 and previous config saved to /var/cache/conftool/dbconfig/20240411-012110-arnaudb.json
• 01:06 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2170', diff saved to https://phabricator.wikimedia.org/P60329 and previous config saved to /var/cache/conftool/dbconfig/20240411-010601-arnaudb.json
• 00:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2170 (T360332)', diff saved to https://phabricator.wikimedia.org/P60328 and previous config saved to /var/cache/conftool/dbconfig/20240411-005054-arnaudb.json
• 00:48 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2170 (T360332)', diff saved to https://phabricator.wikimedia.org/P60327 and previous config saved to /var/cache/conftool/dbconfig/20240411-004758-arnaudb.json
• 00:47 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2170.codfw.wmnet with reason: Maintenance
• 00:47 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2170.codfw.wmnet with reason: Maintenance
• 00:47 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2153 (T360332)', diff saved to https://phabricator.wikimedia.org/P60326 and previous config saved to /var/cache/conftool/dbconfig/20240411-004735-arnaudb.json
• 00:45 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1244 (T356166)', diff saved to https://phabricator.wikimedia.org/P60325 and previous config saved to /var/cache/conftool/dbconfig/20240411-004536-marostegui.json
• 00:45 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1244.eqiad.wmnet with reason: Maintenance
• 00:45 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1244.eqiad.wmnet with reason: Maintenance
• 00:45 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1243 (T356166)', diff saved to https://phabricator.wikimedia.org/P60324 and previous config saved to /var/cache/conftool/dbconfig/20240411-004514-marostegui.json
• 00:32 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P60323 and previous config saved to /var/cache/conftool/dbconfig/20240411-003226-arnaudb.json
• 00:30 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1243', diff saved to https://phabricator.wikimedia.org/P60322 and previous config saved to /var/cache/conftool/dbconfig/20240411-003005-marostegui.json
• 00:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P60321 and previous config saved to /var/cache/conftool/dbconfig/20240411-001718-arnaudb.json
• 00:14 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1243', diff saved to https://phabricator.wikimedia.org/P60320 and previous config saved to /var/cache/conftool/dbconfig/20240411-001458-marostegui.json
• 00:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2153 (T360332)', diff saved to https://phabricator.wikimedia.org/P60319 and previous config saved to /var/cache/conftool/dbconfig/20240411-000211-arnaudb.json

2024-04-10

• 23:59 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1243 (T356166)', diff saved to https://phabricator.wikimedia.org/P60318 and previous config saved to /var/cache/conftool/dbconfig/20240410-235950-marostegui.json
• 23:59 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2153 (T360332)', diff saved to https://phabricator.wikimedia.org/P60317 and previous config saved to /var/cache/conftool/dbconfig/20240410-235920-arnaudb.json
• 23:59 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2153.codfw.wmnet with reason: Maintenance
• 23:59 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2153.codfw.wmnet with reason: Maintenance
• 23:58 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2146 (T360332)', diff saved to https://phabricator.wikimedia.org/P60316 and previous config saved to /var/cache/conftool/dbconfig/20240410-235857-arnaudb.json
• 23:43 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2146', diff saved to https://phabricator.wikimedia.org/P60315 and previous config saved to /var/cache/conftool/dbconfig/20240410-234350-arnaudb.json
• 23:28 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2146', diff saved to https://phabricator.wikimedia.org/P60314 and previous config saved to /var/cache/conftool/dbconfig/20240410-232842-arnaudb.json
• 23:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2146 (T360332)', diff saved to https://phabricator.wikimedia.org/P60313 and previous config saved to /var/cache/conftool/dbconfig/20240410-231335-arnaudb.json
• 23:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2146 (T360332)', diff saved to https://phabricator.wikimedia.org/P60312 and previous config saved to /var/cache/conftool/dbconfig/20240410-231032-arnaudb.json
• 23:10 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2146.codfw.wmnet with reason: Maintenance
• 23:10 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2146.codfw.wmnet with reason: Maintenance
• 23:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2145 (T360332)', diff saved to https://phabricator.wikimedia.org/P60311 and previous config saved to /var/cache/conftool/dbconfig/20240410-231008-arnaudb.json
• 22:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2145', diff saved to https://phabricator.wikimedia.org/P60310 and previous config saved to /var/cache/conftool/dbconfig/20240410-225500-arnaudb.json
• 22:39 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2145', diff saved to https://phabricator.wikimedia.org/P60309 and previous config saved to /var/cache/conftool/dbconfig/20240410-223953-arnaudb.json
• 22:24 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2145 (T360332)', diff saved to https://phabricator.wikimedia.org/P60308 and previous config saved to /var/cache/conftool/dbconfig/20240410-222445-arnaudb.json
• 22:21 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2145 (T360332)', diff saved to https://phabricator.wikimedia.org/P60307 and previous config saved to /var/cache/conftool/dbconfig/20240410-222150-arnaudb.json
• 22:21 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2145.codfw.wmnet with reason: Maintenance
• 22:21 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2145.codfw.wmnet with reason: Maintenance
• 22:20 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2141.codfw.wmnet with reason: Maintenance
• 22:20 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2141.codfw.wmnet with reason: Maintenance
• 22:20 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2130 (T360332)', diff saved to https://phabricator.wikimedia.org/P60306 and previous config saved to /var/cache/conftool/dbconfig/20240410-222028-arnaudb.json
• 22:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2130', diff saved to https://phabricator.wikimedia.org/P60305 and previous config saved to /var/cache/conftool/dbconfig/20240410-220521-arnaudb.json
• 21:56 mutante: prometheus - recreating deleted TLS certs/keys in private repo
• 21:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2130', diff saved to https://phabricator.wikimedia.org/P60304 and previous config saved to /var/cache/conftool/dbconfig/20240410-215014-arnaudb.json
• 21:35 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2130 (T360332)', diff saved to https://phabricator.wikimedia.org/P60303 and previous config saved to /var/cache/conftool/dbconfig/20240410-213506-arnaudb.json
• 21:32 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2130 (T360332)', diff saved to https://phabricator.wikimedia.org/P60302 and previous config saved to /var/cache/conftool/dbconfig/20240410-213203-arnaudb.json
• 21:31 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2130.codfw.wmnet with reason: Maintenance
• 21:31 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2130.codfw.wmnet with reason: Maintenance
• 21:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2116 (T360332)', diff saved to https://phabricator.wikimedia.org/P60301 and previous config saved to /var/cache/conftool/dbconfig/20240410-213140-arnaudb.json
• 21:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2116', diff saved to https://phabricator.wikimedia.org/P60300 and previous config saved to /var/cache/conftool/dbconfig/20240410-211632-arnaudb.json
• 21:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2116', diff saved to https://phabricator.wikimedia.org/P60298 and previous config saved to /var/cache/conftool/dbconfig/20240410-210125-arnaudb.json
• 20:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2116 (T360332)', diff saved to https://phabricator.wikimedia.org/P60297 and previous config saved to /var/cache/conftool/dbconfig/20240410-204617-arnaudb.json
• 20:43 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2116 (T360332)', diff saved to https://phabricator.wikimedia.org/P60296 and previous config saved to /var/cache/conftool/dbconfig/20240410-204316-arnaudb.json
• 20:43 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2116.codfw.wmnet with reason: Maintenance
• 20:42 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2116.codfw.wmnet with reason: Maintenance
• 20:42 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2112 (T360332)', diff saved to https://phabricator.wikimedia.org/P60295 and previous config saved to /var/cache/conftool/dbconfig/20240410-204253-arnaudb.json
• 20:27 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2112', diff saved to https://phabricator.wikimedia.org/P60294 and previous config saved to /var/cache/conftool/dbconfig/20240410-202745-arnaudb.json
• 20:17 cjming: end of UTC late backport window
• 20:15 cjming@deploy1002: Finished scap: Backport for LogStash: log HtmlOutputRendererHelper channel (T356157) (duration: 13m 51s)
• 20:12 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2112', diff saved to https://phabricator.wikimedia.org/P60293 and previous config saved to /var/cache/conftool/dbconfig/20240410-201237-arnaudb.json
• 20:04 cjming@deploy1002: cjming and daniel: Continuing with sync
• 20:04 cjming@deploy1002: cjming and daniel: Backport for LogStash: log HtmlOutputRendererHelper channel (T356157) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:01 cjming@deploy1002: Started scap: Backport for LogStash: log HtmlOutputRendererHelper channel (T356157)
• 19:57 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2112 (T360332)', diff saved to https://phabricator.wikimedia.org/P60292 and previous config saved to /var/cache/conftool/dbconfig/20240410-195730-arnaudb.json
• 19:54 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2112 (T360332)', diff saved to https://phabricator.wikimedia.org/P60291 and previous config saved to /var/cache/conftool/dbconfig/20240410-195430-arnaudb.json
• 19:54 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2112.codfw.wmnet with reason: Maintenance
• 19:54 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2112.codfw.wmnet with reason: Maintenance
• 19:53 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2103.codfw.wmnet with reason: Maintenance
• 19:53 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2103.codfw.wmnet with reason: Maintenance
• 19:52 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2102.codfw.wmnet with reason: Maintenance
• 19:51 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2102.codfw.wmnet with reason: Maintenance
• 19:51 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
• 19:50 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
• 19:50 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1240.eqiad.wmnet with reason: Maintenance
• 19:50 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1240.eqiad.wmnet with reason: Maintenance
• 19:49 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1239.eqiad.wmnet with reason: Maintenance
• 19:49 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1239.eqiad.wmnet with reason: Maintenance
• 19:49 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1235 (T360332)', diff saved to https://phabricator.wikimedia.org/P60290 and previous config saved to /var/cache/conftool/dbconfig/20240410-194909-arnaudb.json
• 19:34 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1235', diff saved to https://phabricator.wikimedia.org/P60289 and previous config saved to /var/cache/conftool/dbconfig/20240410-193402-arnaudb.json
• 19:24 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp3071.esams.wmnet,service=(cdn|ats-be)
• 19:20 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3071.esams.wmnet with OS bullseye
• 19:18 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1235', diff saved to https://phabricator.wikimedia.org/P60288 and previous config saved to /var/cache/conftool/dbconfig/20240410-191854-arnaudb.json
• 19:03 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1235 (T360332)', diff saved to https://phabricator.wikimedia.org/P60287 and previous config saved to /var/cache/conftool/dbconfig/20240410-190347-arnaudb.json
• 18:54 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3071.esams.wmnet with reason: host reimage
• 18:51 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3071.esams.wmnet with reason: host reimage
• 18:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1235 (T360332)', diff saved to https://phabricator.wikimedia.org/P60285 and previous config saved to /var/cache/conftool/dbconfig/20240410-184656-arnaudb.json
• 18:46 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1235.eqiad.wmnet with reason: Maintenance
• 18:46 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1235.eqiad.wmnet with reason: Maintenance
• 18:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1234 (T360332)', diff saved to https://phabricator.wikimedia.org/P60284 and previous config saved to /var/cache/conftool/dbconfig/20240410-184633-arnaudb.json
• 18:34 eevans@deploy1002: helmfile [staging] DONE helmfile.d/services/echostore: apply
• 18:34 eevans@deploy1002: helmfile [staging] START helmfile.d/services/echostore: apply
• 18:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1234', diff saved to https://phabricator.wikimedia.org/P60283 and previous config saved to /var/cache/conftool/dbconfig/20240410-183126-arnaudb.json
• 18:30 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1115.eqiad.wmnet,service=(cdn|ats-be)
• 18:28 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp3071.esams.wmnet with OS bullseye
• 18:26 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1115.eqiad.wmnet with OS bullseye
• 18:24 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp3071.esams.wmnet,service=(cdn|ats-be)
• 18:17 eevans@deploy1002: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
• 18:16 eevans@deploy1002: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
• 18:16 eevans@deploy1002: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
• 18:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1234', diff saved to https://phabricator.wikimedia.org/P60282 and previous config saved to /var/cache/conftool/dbconfig/20240410-181618-arnaudb.json
• 18:15 eevans@deploy1002: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
• 18:08 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1115.eqiad.wmnet with reason: host reimage
• 18:05 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1115.eqiad.wmnet with reason: host reimage
• 18:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1234 (T360332)', diff saved to https://phabricator.wikimedia.org/P60281 and previous config saved to /var/cache/conftool/dbconfig/20240410-180111-arnaudb.json
• 17:58 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1234 (T360332)', diff saved to https://phabricator.wikimedia.org/P60280 and previous config saved to /var/cache/conftool/dbconfig/20240410-175816-arnaudb.json
• 17:58 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1234.eqiad.wmnet with reason: Maintenance
• 17:58 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1234.eqiad.wmnet with reason: Maintenance
• 17:57 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1232 (T360332)', diff saved to https://phabricator.wikimedia.org/P60279 and previous config saved to /var/cache/conftool/dbconfig/20240410-175752-arnaudb.json
• 17:48 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp1115.eqiad.wmnet with OS bullseye
• 17:48 sukhe@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp1115.eqiad.wmnet with OS bullseye
• 17:46 swfrench-wmf: finished updating A:conf hosts to etcd-mirror 0.0.11-1 (T358636)
• 17:42 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1232', diff saved to https://phabricator.wikimedia.org/P60278 and previous config saved to /var/cache/conftool/dbconfig/20240410-174244-arnaudb.json
• 17:37 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp1115.eqiad.wmnet with OS bullseye
• 17:37 swfrench-wmf: restarting etcd-mirror on conf2005.codfw.wmnet for T358636
• 17:35 sukhe@cumin1002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp1115.eqiad.wmnet
• 17:34 sukhe@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp1115.eqiad.wmnet
• 17:27 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1232', diff saved to https://phabricator.wikimedia.org/P60277 and previous config saved to /var/cache/conftool/dbconfig/20240410-172736-arnaudb.json
• 17:21 hashar@deploy1002: Finished scap: Backport for TitleLibrary: Don't register external titles as dependencies (T362222) (duration: 18m 53s)
• 17:14 sukhe@cumin1002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp1115.eqiad.wmnet
• 17:12 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1232 (T360332)', diff saved to https://phabricator.wikimedia.org/P60276 and previous config saved to /var/cache/conftool/dbconfig/20240410-171229-arnaudb.json
• 17:09 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1232 (T360332)', diff saved to https://phabricator.wikimedia.org/P60275 and previous config saved to /var/cache/conftool/dbconfig/20240410-170930-arnaudb.json
• 17:09 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1232.eqiad.wmnet with reason: Maintenance
• 17:09 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1232.eqiad.wmnet with reason: Maintenance
• 17:09 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1228 (T360332)', diff saved to https://phabricator.wikimedia.org/P60274 and previous config saved to /var/cache/conftool/dbconfig/20240410-170907-arnaudb.json
• 17:07 hashar@deploy1002: hashar: Continuing with sync
• 17:07 hashar@deploy1002: hashar: Backport for TitleLibrary: Don't register external titles as dependencies (T362222) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 17:06 sukhe@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp1115.eqiad.wmnet
• 17:06 sukhe@cumin1002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp1115.eqiad.wmnet
• 17:05 sukhe@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp1115.eqiad.wmnet
• 17:05 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp1115.eqiad.wmnet,service=(cdn|ats-be)
• 17:04 sukhe: depool cp1115 for firmware downgrade for PXE boot testing: T350179
• 17:04 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 17:04 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 17:04 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 17:03 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 17:02 hnowlan: killing long-running videoscaler ffmpegs
• 17:02 hashar@deploy1002: Started scap: Backport for TitleLibrary: Don't register external titles as dependencies (T362222)
• 16:54 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1228', diff saved to https://phabricator.wikimedia.org/P60272 and previous config saved to /var/cache/conftool/dbconfig/20240410-165359-arnaudb.json
• 16:50 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 16:50 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 16:50 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 16:50 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 16:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1228', diff saved to https://phabricator.wikimedia.org/P60270 and previous config saved to /var/cache/conftool/dbconfig/20240410-163851-arnaudb.json
• 16:23 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1228 (T360332)', diff saved to https://phabricator.wikimedia.org/P60269 and previous config saved to /var/cache/conftool/dbconfig/20240410-162344-arnaudb.json
• 16:21 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1228 (T360332)', diff saved to https://phabricator.wikimedia.org/P60268 and previous config saved to /var/cache/conftool/dbconfig/20240410-162101-arnaudb.json
• 16:20 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1228.eqiad.wmnet with reason: Maintenance
• 16:20 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1228.eqiad.wmnet with reason: Maintenance
• 16:20 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1219 (T360332)', diff saved to https://phabricator.wikimedia.org/P60267 and previous config saved to /var/cache/conftool/dbconfig/20240410-162039-arnaudb.json
• 16:19 elukey@deploy1002: helmfile [staging] DONE helmfile.d/services/sessionstore: sync
• 16:19 elukey@deploy1002: helmfile [staging] START helmfile.d/services/sessionstore: sync
• 16:16 logmsgbot: lucaswerkmeister-wmde@deploy1002 helmfile [eqiad] DONE helmfile.d/services/termbox: apply
• 16:16 logmsgbot: lucaswerkmeister-wmde@deploy1002 helmfile [eqiad] START helmfile.d/services/termbox: apply
• 16:15 logmsgbot: lucaswerkmeister-wmde@deploy1002 helmfile [codfw] DONE helmfile.d/services/termbox: apply
• 16:15 logmsgbot: lucaswerkmeister-wmde@deploy1002 helmfile [codfw] START helmfile.d/services/termbox: apply
• 16:14 logmsgbot: lucaswerkmeister-wmde@deploy1002 helmfile [staging] DONE helmfile.d/services/termbox: apply
• 16:13 logmsgbot: lucaswerkmeister-wmde@deploy1002 helmfile [staging] START helmfile.d/services/termbox: apply
• 16:12 swfrench-wmf: uploaded etcd-mirror 0.0.11-1 to apt.wikimedia.org (T358636)
• 16:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P60265 and previous config saved to /var/cache/conftool/dbconfig/20240410-160531-arnaudb.json
• 15:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P60264 and previous config saved to /var/cache/conftool/dbconfig/20240410-155024-arnaudb.json
• 15:35 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1219 (T360332)', diff saved to https://phabricator.wikimedia.org/P60262 and previous config saved to /var/cache/conftool/dbconfig/20240410-153516-arnaudb.json
• 15:32 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1219 (T360332)', diff saved to https://phabricator.wikimedia.org/P60261 and previous config saved to /var/cache/conftool/dbconfig/20240410-153229-arnaudb.json
• 15:32 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1219.eqiad.wmnet with reason: Maintenance
• 15:32 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1219.eqiad.wmnet with reason: Maintenance
• 15:32 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1218 (T360332)', diff saved to https://phabricator.wikimedia.org/P60260 and previous config saved to /var/cache/conftool/dbconfig/20240410-153207-arnaudb.json
• 15:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1218', diff saved to https://phabricator.wikimedia.org/P60259 and previous config saved to /var/cache/conftool/dbconfig/20240410-151659-arnaudb.json
• 15:14 hnowlan@deploy1002: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
• 15:14 hnowlan@deploy1002: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
• 15:14 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
• 15:13 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
• 15:03 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1243 (T356166)', diff saved to https://phabricator.wikimedia.org/P60258 and previous config saved to /var/cache/conftool/dbconfig/20240410-150327-marostegui.json
• 15:03 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1243.eqiad.wmnet with reason: Maintenance
• 15:03 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1243.eqiad.wmnet with reason: Maintenance
• 15:03 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1242 (T356166)', diff saved to https://phabricator.wikimedia.org/P60257 and previous config saved to /var/cache/conftool/dbconfig/20240410-150304-marostegui.json
• 15:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1218', diff saved to https://phabricator.wikimedia.org/P60256 and previous config saved to /var/cache/conftool/dbconfig/20240410-150152-arnaudb.json
• 14:58 moritzm: installing debian-archive-keyring updates on buster
• 14:55 akosiaris: kill all ffmpegs on mw1437 and increase weight of mw1347 from 10 to 30 to direct most queries to it while the other 3 videoscalers serve the backlog
• 14:54 akosiaris@cumin1002: conftool action : set/weight=30; selector: name=mw1437.*.wmnet,dc=eqiad
• 14:51 hnowlan@deploy1002: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
• 14:51 hnowlan@deploy1002: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
• 14:50 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
• 14:50 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
• 14:47 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1242', diff saved to https://phabricator.wikimedia.org/P60255 and previous config saved to /var/cache/conftool/dbconfig/20240410-144757-marostegui.json
• 14:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1218 (T360332)', diff saved to https://phabricator.wikimedia.org/P60254 and previous config saved to /var/cache/conftool/dbconfig/20240410-144644-arnaudb.json
• 14:44 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1218 (T360332)', diff saved to https://phabricator.wikimedia.org/P60253 and previous config saved to /var/cache/conftool/dbconfig/20240410-144400-arnaudb.json
• 14:43 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1218.eqiad.wmnet with reason: Maintenance
• 14:43 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1218.eqiad.wmnet with reason: Maintenance
• 14:43 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1207 (T360332)', diff saved to https://phabricator.wikimedia.org/P60252 and previous config saved to /var/cache/conftool/dbconfig/20240410-144336-arnaudb.json
• 14:32 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1242', diff saved to https://phabricator.wikimedia.org/P60251 and previous config saved to /var/cache/conftool/dbconfig/20240410-143249-marostegui.json
• 14:28 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P60250 and previous config saved to /var/cache/conftool/dbconfig/20240410-142829-arnaudb.json
• 14:21 sukhe@cumin1002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cp4052.ulsfo.wmnet
• 14:20 sukhe@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp4052.ulsfo.wmnet
• 14:17 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1242 (T356166)', diff saved to https://phabricator.wikimedia.org/P60249 and previous config saved to /var/cache/conftool/dbconfig/20240410-141742-marostegui.json
• 14:17 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1112.eqiad.wmnet,service=(cdn|ats-be)
• 14:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P60248 and previous config saved to /var/cache/conftool/dbconfig/20240410-141322-arnaudb.json
• 14:07 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1112.eqiad.wmnet with OS bullseye
• 13:58 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp4052.ulsfo.wmnet,service=(cdn|ats-be)
• 13:58 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1207 (T360332)', diff saved to https://phabricator.wikimedia.org/P60246 and previous config saved to /var/cache/conftool/dbconfig/20240410-135814-arnaudb.json
• 13:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1207 (T360332)', diff saved to https://phabricator.wikimedia.org/P60245 and previous config saved to /var/cache/conftool/dbconfig/20240410-135525-arnaudb.json
• 13:55 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1207.eqiad.wmnet with reason: Maintenance
• 13:55 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1207.eqiad.wmnet with reason: Maintenance
• 13:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1206 (T360332)', diff saved to https://phabricator.wikimedia.org/P60244 and previous config saved to /var/cache/conftool/dbconfig/20240410-135502-arnaudb.json
• 13:54 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4052.ulsfo.wmnet with OS bullseye
• 13:49 denisse: Delete unused Prometheus TLS certificates - T360414
• 13:47 moritzm: installing unbound security updates
• 13:46 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1112.eqiad.wmnet with reason: host reimage
• 13:43 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1112.eqiad.wmnet with reason: host reimage
• 13:39 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1206', diff saved to https://phabricator.wikimedia.org/P60243 and previous config saved to /var/cache/conftool/dbconfig/20240410-133955-arnaudb.json
• 13:39 eevans@deploy1002: helmfile [staging] DONE helmfile.d/services/sessionstore: apply
• 13:33 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage
• 13:30 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 30 days, 0:00:00 on elastic2088.codfw.wmnet with reason: T361525
• 13:30 bking@cumin2002: START - Cookbook sre.hosts.downtime for 30 days, 0:00:00 on elastic2088.codfw.wmnet with reason: T361525
• 13:30 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage
• 13:28 eevans@deploy1002: helmfile [staging] START helmfile.d/services/sessionstore: apply
• 13:27 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp1112.eqiad.wmnet with OS bullseye
• 13:26 sukhe@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp1112.eqiad.wmnet with OS bullseye
• 13:24 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1206', diff saved to https://phabricator.wikimedia.org/P60242 and previous config saved to /var/cache/conftool/dbconfig/20240410-132447-arnaudb.json
• 13:17 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1242 (T356166)', diff saved to https://phabricator.wikimedia.org/P60241 and previous config saved to /var/cache/conftool/dbconfig/20240410-131716-marostegui.json
• 13:17 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1242.eqiad.wmnet with reason: Maintenance
• 13:17 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp1112.eqiad.wmnet with OS bullseye
• 13:16 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1242.eqiad.wmnet with reason: Maintenance
• 13:16 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1241 (T356166)', diff saved to https://phabricator.wikimedia.org/P60240 and previous config saved to /var/cache/conftool/dbconfig/20240410-131653-marostegui.json
• 13:16 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp1112.eqiad.wmnet,service=(cdn|ats-be)
• 13:09 volans@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 13:09 volans@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: test restoring dns entry - volans@cumin2002"
• 13:09 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1206 (T360332)', diff saved to https://phabricator.wikimedia.org/P60239 and previous config saved to /var/cache/conftool/dbconfig/20240410-130940-arnaudb.json
• 13:09 volans@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: test restoring dns entry - volans@cumin2002"
• 13:07 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp4052.ulsfo.wmnet with OS bullseye
• 13:07 volans@cumin2002: START - Cookbook sre.dns.netbox
• 13:07 sukhe: depool cp4052 for PXE boot issue testing
• 13:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1206 (T360332)', diff saved to https://phabricator.wikimedia.org/P60238 and previous config saved to /var/cache/conftool/dbconfig/20240410-130650-arnaudb.json
• 13:07 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp4052.ulsfo.wmnet,service=(cdn|ats-be)
• 13:06 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1206.eqiad.wmnet with reason: Maintenance
• 13:06 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1206.eqiad.wmnet with reason: Maintenance
• 13:06 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1196 (T360332)', diff saved to https://phabricator.wikimedia.org/P60237 and previous config saved to /var/cache/conftool/dbconfig/20240410-130626-arnaudb.json
• 13:05 volans@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 13:05 volans@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: test removing dns entry - volans@cumin2002"
• 13:05 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=dns6001.wikimedia.org,service=authdns-update
• 13:04 volans@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: test removing dns entry - volans@cumin2002"
• 13:02 volans@cumin2002: START - Cookbook sre.dns.netbox
• 13:01 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1241', diff saved to https://phabricator.wikimedia.org/P60236 and previous config saved to /var/cache/conftool/dbconfig/20240410-130145-marostegui.json
• 12:59 slyngshede@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2003.wikimedia.org
• 12:59 slyngshede@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:59 slyngshede@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: idp-test2003.wikimedia.org decommissioned, removing all IPs except the asset tag one - slyngshede@cumin1002"
• 12:56 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=dns6001.wikimedia.org,service=authdns-update
• 12:56 slyngshede@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: idp-test2003.wikimedia.org decommissioned, removing all IPs except the asset tag one - slyngshede@cumin1002"
• 12:53 slyngshede@cumin1002: START - Cookbook sre.dns.netbox
• 12:51 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P60235 and previous config saved to /var/cache/conftool/dbconfig/20240410-125119-arnaudb.json
• 12:48 slyngshede@cumin1002: START - Cookbook sre.hosts.decommission for hosts idp-test2003.wikimedia.org
• 12:46 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1241', diff saved to https://phabricator.wikimedia.org/P60234 and previous config saved to /var/cache/conftool/dbconfig/20240410-124638-marostegui.json
• 12:45 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "update for latest VMs - jmm@cumin2002"
• 12:44 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "update for latest VMs - jmm@cumin2002"
• 12:25 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 12:25 marostegui@cumin1002: dbctl commit (dc=all): 'db1175 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P60231 and previous config saved to /var/cache/conftool/dbconfig/20240410-122518-root.json
• 12:25 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 12:21 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1196 (T360332)', diff saved to https://phabricator.wikimedia.org/P60230 and previous config saved to /var/cache/conftool/dbconfig/20240410-122104-arnaudb.json
• 12:20 slyngshede@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on idp-test1002.wikimedia.org with reason: host reimage
• 12:18 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1196 (T360332)', diff saved to https://phabricator.wikimedia.org/P60229 and previous config saved to /var/cache/conftool/dbconfig/20240410-121814-arnaudb.json
• 12:18 slyngshede@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on idp-test1002.wikimedia.org with reason: host reimage
• 12:18 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
• 12:18 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
• 12:18 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1196.eqiad.wmnet with reason: Maintenance
• 12:17 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1196.eqiad.wmnet with reason: Maintenance
• 12:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1186 (T360332)', diff saved to https://phabricator.wikimedia.org/P60228 and previous config saved to /var/cache/conftool/dbconfig/20240410-121743-arnaudb.json
• 12:15 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ mwscript extensions/Wikibase/repo/maintenance/changePropertyDataType.php wikidatawiki --property-id P4496 --new-data-type external-id --summary 'T359297' # succeeded
• 12:14 Lucas_WMDE: lucaswerkmeister-wmde@deploy1002 ~ $ mwscript-k8s extensions/Wikibase/repo/maintenance/changePropertyDataType.php wikidatawiki --property-id P4496 --new-data-type external-id --summary 'T359297' # failed, will retry with non-k8s mwscript
• 12:12 cgoubert@cumin1002: conftool action : set/weight=10:pooled=yes; selector: name=(mw1421.eqiad.wmnet|mw1422.eqiad.wmnet|mw1491.eqiad.wmnet|mw1492.eqiad.wmnet|mw1493.eqiad.wmnet),cluster=kubernetes,service=kubesvc
• 12:11 claime: Pooling and uncordoning mw1421.eqiad.wmnet,mw1422.eqiad.wmnet,mw1491.eqiad.wmnet,mw1492.eqiad.wmnet,mw1493.eqiad.wmnet - T351074
• 12:10 marostegui@cumin1002: dbctl commit (dc=all): 'db1175 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P60227 and previous config saved to /var/cache/conftool/dbconfig/20240410-121012-root.json
• 12:04 slyngshede@cumin1002: START - Cookbook sre.hosts.reimage for host idp-test1002.wikimedia.org with OS bookworm
• 12:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P60226 and previous config saved to /var/cache/conftool/dbconfig/20240410-120235-arnaudb.json
• 12:01 claime: Running homer 'cr*eqiad*' commit 'T351074' and homer 'lsw1-e3-eqiad*' commit 'T351074'
• 11:55 marostegui@cumin1002: dbctl commit (dc=all): 'db1175 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P60225 and previous config saved to /var/cache/conftool/dbconfig/20240410-115506-root.json
• 11:54 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1492.eqiad.wmnet with OS bullseye
• 11:53 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1491.eqiad.wmnet with OS bullseye
• 11:49 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1422.eqiad.wmnet with OS bullseye
• 11:47 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P60224 and previous config saved to /var/cache/conftool/dbconfig/20240410-114728-arnaudb.json
• 11:45 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1493.eqiad.wmnet with OS bullseye
• 11:42 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1421.eqiad.wmnet with OS bullseye
• 11:40 marostegui@cumin1002: dbctl commit (dc=all): 'db1175 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P60223 and previous config saved to /var/cache/conftool/dbconfig/20240410-114001-root.json
• 11:38 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1492.eqiad.wmnet with reason: host reimage
• 11:34 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1491.eqiad.wmnet with reason: host reimage
• 11:32 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1186 (T360332)', diff saved to https://phabricator.wikimedia.org/P60222 and previous config saved to /var/cache/conftool/dbconfig/20240410-113220-arnaudb.json
• 11:31 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1422.eqiad.wmnet with reason: host reimage
• 11:29 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1186 (T360332)', diff saved to https://phabricator.wikimedia.org/P60221 and previous config saved to /var/cache/conftool/dbconfig/20240410-112929-arnaudb.json
• 11:29 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1186.eqiad.wmnet with reason: Maintenance
• 11:29 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1186.eqiad.wmnet with reason: Maintenance
• 11:29 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1169 (T360332)', diff saved to https://phabricator.wikimedia.org/P60220 and previous config saved to /var/cache/conftool/dbconfig/20240410-112907-arnaudb.json
• 11:28 jiji@deploy1002: Finished scap: Deploy chart changes in gerrit:1015342 (duration: 08m 18s)
• 11:27 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1493.eqiad.wmnet with reason: host reimage
• 11:24 marostegui@cumin1002: dbctl commit (dc=all): 'db1175 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P60219 and previous config saved to /var/cache/conftool/dbconfig/20240410-112455-root.json
• 11:24 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1421.eqiad.wmnet with reason: host reimage
• 11:23 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw1493.eqiad.wmnet with reason: host reimage
• 11:22 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw1492.eqiad.wmnet with reason: host reimage
• 11:22 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw1491.eqiad.wmnet with reason: host reimage
• 11:21 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw1422.eqiad.wmnet with reason: host reimage
• 11:21 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw1421.eqiad.wmnet with reason: host reimage
• 11:19 jiji@deploy1002: Started scap: Deploy chart changes in gerrit:1015342
• 11:16 mvolz@deploy1002: helmfile [eqiad] DONE helmfile.d/services/citoid: apply
• 11:15 mvolz@deploy1002: helmfile [eqiad] START helmfile.d/services/citoid: apply
• 11:14 mvolz@deploy1002: helmfile [codfw] DONE helmfile.d/services/citoid: apply
• 11:14 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P60218 and previous config saved to /var/cache/conftool/dbconfig/20240410-111400-arnaudb.json
• 11:13 mvolz@deploy1002: helmfile [codfw] START helmfile.d/services/citoid: apply
• 11:12 mvolz@deploy1002: helmfile [staging] DONE helmfile.d/services/citoid: apply
• 11:12 mvolz@deploy1002: helmfile [staging] START helmfile.d/services/citoid: apply
• 11:10 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw1493.eqiad.wmnet with OS bullseye
• 11:09 marostegui@cumin1002: dbctl commit (dc=all): 'db1175 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P60217 and previous config saved to /var/cache/conftool/dbconfig/20240410-110949-root.json
• 11:09 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw1492.eqiad.wmnet with OS bullseye
• 11:09 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw1491.eqiad.wmnet with OS bullseye
• 11:08 jiji@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
• 11:08 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw1422.eqiad.wmnet with OS bullseye
• 11:08 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw1421.eqiad.wmnet with OS bullseye
• 11:07 jiji@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
• 11:07 jiji@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
• 11:07 jiji@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
• 11:03 jiji@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
• 11:02 jiji@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
• 11:02 jiji@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
• 11:02 jiji@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
• 10:59 claime: Depooling mw1421.eqiad.wmnet,mw1422.eqiad.wmnet,mw1491.eqiad.wmnet,mw1492.eqiad.wmnet,mw1493.eqiad.wmnet - T351074
• 10:58 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P60216 and previous config saved to /var/cache/conftool/dbconfig/20240410-105852-arnaudb.json
• 10:56 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1175.eqiad.wmnet with OS bookworm
• 10:54 marostegui@cumin1002: dbctl commit (dc=all): 'db1175 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P60215 and previous config saved to /var/cache/conftool/dbconfig/20240410-105444-root.json
• 10:53 jiji@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
• 10:53 jiji@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
• 10:43 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1169 (T360332)', diff saved to https://phabricator.wikimedia.org/P60214 and previous config saved to /var/cache/conftool/dbconfig/20240410-104345-arnaudb.json
• 10:40 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1169 (T360332)', diff saved to https://phabricator.wikimedia.org/P60213 and previous config saved to /var/cache/conftool/dbconfig/20240410-104053-arnaudb.json
• 10:40 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1169.eqiad.wmnet with reason: Maintenance
• 10:40 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1169.eqiad.wmnet with reason: Maintenance
• 10:40 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1163 (T360332)', diff saved to https://phabricator.wikimedia.org/P60212 and previous config saved to /var/cache/conftool/dbconfig/20240410-104030-arnaudb.json
• 10:35 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1175.eqiad.wmnet with reason: host reimage
• 10:32 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1175.eqiad.wmnet with reason: host reimage
• 10:25 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1163', diff saved to https://phabricator.wikimedia.org/P60211 and previous config saved to /var/cache/conftool/dbconfig/20240410-102523-arnaudb.json
• 10:21 claime: Enabling and running puppet on O:docker_registry_ha::registry - T360636
• 10:19 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db1175.eqiad.wmnet with OS bookworm
• 10:18 claime: Enabling and running puppet on registry1003.eqiad.wmnet - T360636
• 10:17 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1175 T362036', diff saved to https://phabricator.wikimedia.org/P60210 and previous config saved to /var/cache/conftool/dbconfig/20240410-101746-root.json
• 10:16 claime: Disabling puppet on O:docker_registry_ha::registry - T360636
• 10:12 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-role (exit_code=0) for role: mariadb::sanitarium_master
• 10:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1163', diff saved to https://phabricator.wikimedia.org/P60209 and previous config saved to /var/cache/conftool/dbconfig/20240410-101015-arnaudb.json
• 10:08 jiji@deploy1002: Finished scap: (no justification provided) (duration: 27m 59s)
• 09:58 jmm@cumin2002: START - Cookbook sre.puppet.migrate-role for role: mariadb::sanitarium_master
• 09:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1163 (T360332)', diff saved to https://phabricator.wikimedia.org/P60208 and previous config saved to /var/cache/conftool/dbconfig/20240410-095508-arnaudb.json
• 09:52 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1163 (T360332)', diff saved to https://phabricator.wikimedia.org/P60207 and previous config saved to /var/cache/conftool/dbconfig/20240410-095214-arnaudb.json
• 09:52 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1163.eqiad.wmnet with reason: Maintenance
• 09:51 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1163.eqiad.wmnet with reason: Maintenance
• 09:42 effie: running scap sync-world to rebuild mw image and pick up gerrit:1015338
• 09:40 jiji@deploy1002: Started scap: (no justification provided)
• 08:49 fabfur@cumin1002: conftool action : set/pooled=yes; selector: name=cp3070.esams.wmnet
• 08:42 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3070.esams.wmnet with OS bullseye
• 08:38 arnaudb@cumin1002: dbctl commit (dc=all): 'db2212 (re)pooling @ 100%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P60206 and previous config saved to /var/cache/conftool/dbconfig/20240410-083822-arnaudb.json
• 08:35 isaranto@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
• 08:34 gmodena@deploy1002: Finished deploy [airflow-dags/analytics@46818a3]: Deploying cassandra_load_pageview_top_articles changes MR#648 (duration: 00m 33s)
• 08:34 hashar@deploy1002: Synchronized php: group1 wikis to 1.42.0-wmf.26 refs T360158 (duration: 13m 05s)
• 08:34 gmodena@deploy1002: Started deploy [airflow-dags/analytics@46818a3]: Deploying cassandra_load_pageview_top_articles changes MR#648
• 08:25 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 08:25 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 08:25 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 08:25 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 08:24 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 08:24 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 08:23 arnaudb@cumin1002: dbctl commit (dc=all): 'db2212 (re)pooling @ 75%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P60205 and previous config saved to /var/cache/conftool/dbconfig/20240410-082316-arnaudb.json
• 08:21 hashar@deploy1002: rebuilt and synchronized wikiversions files: group1 wikis to 1.42.0-wmf.26 refs T360158
• 08:18 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3070.esams.wmnet with reason: host reimage
• 08:15 fabfur@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3070.esams.wmnet with reason: host reimage
• 08:08 arnaudb@cumin1002: dbctl commit (dc=all): 'db2212 (re)pooling @ 50%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P60204 and previous config saved to /var/cache/conftool/dbconfig/20240410-080810-arnaudb.json
• 07:56 moritzm: installing glibc security updates on bullseye
• 07:53 arnaudb@cumin1002: dbctl commit (dc=all): 'db2212 (re)pooling @ 25%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P60203 and previous config saved to /var/cache/conftool/dbconfig/20240410-075304-arnaudb.json
• 07:52 fabfur@cumin1002: START - Cookbook sre.hosts.reimage for host cp3070.esams.wmnet with OS bullseye
• 07:51 arnaudb@cumin1002: dbctl commit (dc=all): 'db2112 (re)pooling @ 100%: Post clone (src)', diff saved to https://phabricator.wikimedia.org/P60202 and previous config saved to /var/cache/conftool/dbconfig/20240410-075150-arnaudb.json
• 07:50 fabfur@cumin1002: conftool action : set/pooled=no; selector: name=cp3070.esams.wmnet
• 07:37 arnaudb@cumin1002: dbctl commit (dc=all): 'db2212 (re)pooling @ 16%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P60201 and previous config saved to /var/cache/conftool/dbconfig/20240410-073759-arnaudb.json
• 07:36 arnaudb@cumin1002: dbctl commit (dc=all): 'db2112 (re)pooling @ 75%: Post clone (src)', diff saved to https://phabricator.wikimedia.org/P60200 and previous config saved to /var/cache/conftool/dbconfig/20240410-073644-arnaudb.json
• 07:33 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-role (exit_code=0) for role: dumps::generation::server::spare
• 07:29 akosiaris@deploy1002: Synchronized wmf-config/mc.php: Dummy sync for https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1018332 (duration: 14m 03s)
• 07:25 jmm@cumin2002: START - Cookbook sre.puppet.migrate-role for role: dumps::generation::server::spare
• 07:22 arnaudb@cumin1002: dbctl commit (dc=all): 'db2212 (re)pooling @ 8%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P60199 and previous config saved to /var/cache/conftool/dbconfig/20240410-072253-arnaudb.json
• 07:21 arnaudb@cumin1002: dbctl commit (dc=all): 'db2112 (re)pooling @ 50%: Post clone (src)', diff saved to https://phabricator.wikimedia.org/P60198 and previous config saved to /var/cache/conftool/dbconfig/20240410-072137-arnaudb.json
• 07:07 arnaudb@cumin1002: dbctl commit (dc=all): 'db2212 (re)pooling @ 4%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P60197 and previous config saved to /var/cache/conftool/dbconfig/20240410-070745-arnaudb.json
• 07:06 arnaudb@cumin1002: dbctl commit (dc=all): 'db2112 (re)pooling @ 25%: Post clone (src)', diff saved to https://phabricator.wikimedia.org/P60196 and previous config saved to /var/cache/conftool/dbconfig/20240410-070631-arnaudb.json
• 06:59 marostegui@cumin1002: dbctl commit (dc=all): 'db1166 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P60195 and previous config saved to /var/cache/conftool/dbconfig/20240410-065929-root.json
• 06:52 arnaudb@cumin1002: dbctl commit (dc=all): 'db2212 (re)pooling @ 2%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P60194 and previous config saved to /var/cache/conftool/dbconfig/20240410-065239-arnaudb.json
• 06:51 arnaudb@cumin1002: dbctl commit (dc=all): 'db2112 (re)pooling @ 20%: Post clone (src)', diff saved to https://phabricator.wikimedia.org/P60193 and previous config saved to /var/cache/conftool/dbconfig/20240410-065125-arnaudb.json
• 06:44 marostegui@cumin1002: dbctl commit (dc=all): 'db1166 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P60192 and previous config saved to /var/cache/conftool/dbconfig/20240410-064423-root.json
• 06:37 arnaudb@cumin1002: dbctl commit (dc=all): 'db2212 (re)pooling @ 1%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P60191 and previous config saved to /var/cache/conftool/dbconfig/20240410-063734-arnaudb.json
• 06:36 arnaudb@cumin1002: dbctl commit (dc=all): 'db2112 (re)pooling @ 10%: Post clone (src)', diff saved to https://phabricator.wikimedia.org/P60190 and previous config saved to /var/cache/conftool/dbconfig/20240410-063620-arnaudb.json
• 06:29 marostegui@cumin1002: dbctl commit (dc=all): 'db1166 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P60189 and previous config saved to /var/cache/conftool/dbconfig/20240410-062917-root.json
• 06:21 arnaudb@cumin1002: dbctl commit (dc=all): 'db2112 (re)pooling @ 5%: Post clone (src)', diff saved to https://phabricator.wikimedia.org/P60188 and previous config saved to /var/cache/conftool/dbconfig/20240410-062114-arnaudb.json
• 06:20 marostegui@cumin1002: dbctl commit (dc=all): 'db1246 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P60187 and previous config saved to /var/cache/conftool/dbconfig/20240410-062003-root.json
• 06:14 marostegui@cumin1002: dbctl commit (dc=all): 'db1166 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P60186 and previous config saved to /var/cache/conftool/dbconfig/20240410-061411-root.json
• 06:04 marostegui@cumin1002: dbctl commit (dc=all): 'db1246 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P60185 and previous config saved to /var/cache/conftool/dbconfig/20240410-060457-root.json
• 05:59 marostegui@cumin1002: dbctl commit (dc=all): 'db1166 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P60184 and previous config saved to /var/cache/conftool/dbconfig/20240410-055906-root.json
• 05:49 marostegui@cumin1002: dbctl commit (dc=all): 'db1246 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P60183 and previous config saved to /var/cache/conftool/dbconfig/20240410-054952-root.json
• 05:44 marostegui@cumin1002: dbctl commit (dc=all): 'db1166 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P60182 and previous config saved to /var/cache/conftool/dbconfig/20240410-054400-root.json
• 05:34 marostegui@cumin1002: dbctl commit (dc=all): 'db1246 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P60181 and previous config saved to /var/cache/conftool/dbconfig/20240410-053445-root.json
• 05:33 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1166.eqiad.wmnet with OS bookworm
• 05:28 marostegui@cumin1002: dbctl commit (dc=all): 'db1166 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P60180 and previous config saved to /var/cache/conftool/dbconfig/20240410-052854-root.json
• 05:19 marostegui@cumin1002: dbctl commit (dc=all): 'db1246 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P60179 and previous config saved to /var/cache/conftool/dbconfig/20240410-051939-root.json
• 05:12 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1166.eqiad.wmnet with reason: host reimage
• 05:10 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1166.eqiad.wmnet with reason: host reimage
• 05:04 marostegui@cumin1002: dbctl commit (dc=all): 'db1246 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P60178 and previous config saved to /var/cache/conftool/dbconfig/20240410-050434-root.json
• 04:58 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db1166.eqiad.wmnet with OS bookworm
• 04:57 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1166 T362134', diff saved to https://phabricator.wikimedia.org/P60177 and previous config saved to /var/cache/conftool/dbconfig/20240410-045710-marostegui.json
• 04:56 marostegui@cumin1002: dbctl commit (dc=all): 'Repool db1223', diff saved to https://phabricator.wikimedia.org/P60176 and previous config saved to /var/cache/conftool/dbconfig/20240410-045632-marostegui.json
• 04:55 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1223 T362134', diff saved to https://phabricator.wikimedia.org/P60175 and previous config saved to /var/cache/conftool/dbconfig/20240410-045534-marostegui.json
• 04:49 marostegui@cumin1002: dbctl commit (dc=all): 'db1246 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P60174 and previous config saved to /var/cache/conftool/dbconfig/20240410-044928-root.json
• 04:46 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: Kernel reboot
• 04:46 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on 6 hosts with reason: Kernel reboot
• 04:16 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1241 (T356166)', diff saved to https://phabricator.wikimedia.org/P60173 and previous config saved to /var/cache/conftool/dbconfig/20240410-041604-marostegui.json
• 04:15 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1241.eqiad.wmnet with reason: Maintenance
• 04:15 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1241.eqiad.wmnet with reason: Maintenance
• 04:15 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1221 (T356166)', diff saved to https://phabricator.wikimedia.org/P60172 and previous config saved to /var/cache/conftool/dbconfig/20240410-041541-marostegui.json
• 04:00 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P60171 and previous config saved to /var/cache/conftool/dbconfig/20240410-040033-marostegui.json
• 03:45 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P60170 and previous config saved to /var/cache/conftool/dbconfig/20240410-034526-marostegui.json
• 03:30 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1221 (T356166)', diff saved to https://phabricator.wikimedia.org/P60169 and previous config saved to /var/cache/conftool/dbconfig/20240410-033019-marostegui.json

2024-04-09

• 23:17 eileen: config revision changed from 7908b55e to 974afe9c
• 23:08 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2209 (T360332)', diff saved to https://phabricator.wikimedia.org/P60168 and previous config saved to /var/cache/conftool/dbconfig/20240409-230828-arnaudb.json
• 23:08 eileen: config revision changed from 064d18b0 to 7908b55e
• 22:58 eileen: config revision changed from 8fb02f33 to 064d18b0
• 22:53 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2209', diff saved to https://phabricator.wikimedia.org/P60167 and previous config saved to /var/cache/conftool/dbconfig/20240409-225321-arnaudb.json
• 22:51 eileen: config revision changed from df416a50 to 8fb02f33
• 22:48 eileen: config revision changed from cea14e30 to df416a50
• 22:42 eileen: config revision changed from 075ddd44 to cea14e30
• 22:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2209', diff saved to https://phabricator.wikimedia.org/P60166 and previous config saved to /var/cache/conftool/dbconfig/20240409-223813-arnaudb.json
• 22:24 eileen: config revision changed from 3c1a0267 to 4638a4d2
• 22:23 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2209 (T360332)', diff saved to https://phabricator.wikimedia.org/P60165 and previous config saved to /var/cache/conftool/dbconfig/20240409-222306-arnaudb.json
• 22:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2209 (T360332)', diff saved to https://phabricator.wikimedia.org/P60164 and previous config saved to /var/cache/conftool/dbconfig/20240409-220755-arnaudb.json
• 22:07 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2209.codfw.wmnet with reason: Maintenance
• 22:07 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2209.codfw.wmnet with reason: Maintenance
• 22:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2205 (T360332)', diff saved to https://phabricator.wikimedia.org/P60163 and previous config saved to /var/cache/conftool/dbconfig/20240409-220732-arnaudb.json
• 22:03 eileen: civicrm upgraded from b05fd08f to a382a7b0
• 21:52 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2205', diff saved to https://phabricator.wikimedia.org/P60162 and previous config saved to /var/cache/conftool/dbconfig/20240409-215225-arnaudb.json
• 21:38 eileen: civicrm upgraded from 8c7cc208 to b05fd08f
• 21:37 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2205', diff saved to https://phabricator.wikimedia.org/P60161 and previous config saved to /var/cache/conftool/dbconfig/20240409-213717-arnaudb.json
• 21:22 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2205 (T360332)', diff saved to https://phabricator.wikimedia.org/P60160 and previous config saved to /var/cache/conftool/dbconfig/20240409-212210-arnaudb.json
• 21:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2205 (T360332)', diff saved to https://phabricator.wikimedia.org/P60159 and previous config saved to /var/cache/conftool/dbconfig/20240409-210656-arnaudb.json
• 21:07 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2205.codfw.wmnet with reason: Maintenance
• 21:06 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2205.codfw.wmnet with reason: Maintenance
• 21:06 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2194 (T360332)', diff saved to https://phabricator.wikimedia.org/P60158 and previous config saved to /var/cache/conftool/dbconfig/20240409-210633-arnaudb.json
• 21:01 cjming: end of UTC late backport window
• 21:00 cjming@deploy1002: Finished scap: Backport for extension.json: add pagetriage-copyvio right to the highvolume grant (T362188) (duration: 15m 27s)
• 20:51 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2194', diff saved to https://phabricator.wikimedia.org/P60157 and previous config saved to /var/cache/conftool/dbconfig/20240409-205125-arnaudb.json
• 20:48 cjming@deploy1002: jjmc89 and cjming: Continuing with sync
• 20:47 cjming@deploy1002: jjmc89 and cjming: Backport for extension.json: add pagetriage-copyvio right to the highvolume grant (T362188) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:45 cdanis@deploy2002: helmfile [aux-k8s-eqiad] DONE helmfile.d/aus-k8s-eqiad-services/jaeger: apply
• 20:45 cjming@deploy1002: Started scap: Backport for extension.json: add pagetriage-copyvio right to the highvolume grant (T362188)
• 20:44 cdanis@deploy2002: helmfile [aux-k8s-eqiad] START helmfile.d/aus-k8s-eqiad-services/jaeger: apply
• 20:44 cjming@deploy1002: Finished scap: Backport for extension.json: add pagetriage-copyvio right to the highvolume grant (T362188) (duration: 16m 16s)
• 20:36 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2194', diff saved to https://phabricator.wikimedia.org/P60156 and previous config saved to /var/cache/conftool/dbconfig/20240409-203617-arnaudb.json
• 20:32 cjming@deploy1002: jjmc89 and cjming: Continuing with sync
• 20:30 cjming@deploy1002: jjmc89 and cjming: Backport for extension.json: add pagetriage-copyvio right to the highvolume grant (T362188) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:28 cjming@deploy1002: Started scap: Backport for extension.json: add pagetriage-copyvio right to the highvolume grant (T362188)
• 20:24 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2207 (T360332)', diff saved to https://phabricator.wikimedia.org/P60155 and previous config saved to /var/cache/conftool/dbconfig/20240409-202410-arnaudb.json
• 20:21 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2194 (T360332)', diff saved to https://phabricator.wikimedia.org/P60154 and previous config saved to /var/cache/conftool/dbconfig/20240409-202110-arnaudb.json
• 20:09 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P60153 and previous config saved to /var/cache/conftool/dbconfig/20240409-200903-arnaudb.json
• 20:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2194 (T360332)', diff saved to https://phabricator.wikimedia.org/P60152 and previous config saved to /var/cache/conftool/dbconfig/20240409-200556-arnaudb.json
• 20:05 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2194.codfw.wmnet with reason: Maintenance
• 20:05 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2194.codfw.wmnet with reason: Maintenance
• 20:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2190 (T360332)', diff saved to https://phabricator.wikimedia.org/P60151 and previous config saved to /var/cache/conftool/dbconfig/20240409-200533-arnaudb.json
• 20:02 cdanis@deploy2002: helmfile [aux-k8s-eqiad] DONE helmfile.d/aus-k8s-eqiad-services/jaeger: apply
• 20:02 eileen: config revision changed from abccfdc0 to 3c1a0267
• 20:02 cdanis@deploy2002: helmfile [aux-k8s-eqiad] START helmfile.d/aus-k8s-eqiad-services/jaeger: apply
• 19:53 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P60150 and previous config saved to /var/cache/conftool/dbconfig/20240409-195355-arnaudb.json
• 19:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2190', diff saved to https://phabricator.wikimedia.org/P60149 and previous config saved to /var/cache/conftool/dbconfig/20240409-195024-arnaudb.json
• 19:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2207 (T360332)', diff saved to https://phabricator.wikimedia.org/P60148 and previous config saved to /var/cache/conftool/dbconfig/20240409-193848-arnaudb.json
• 19:36 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2207 (T360332)', diff saved to https://phabricator.wikimedia.org/P60147 and previous config saved to /var/cache/conftool/dbconfig/20240409-193611-arnaudb.json
• 19:36 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2207.codfw.wmnet with reason: Maintenance
• 19:35 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2207.codfw.wmnet with reason: Maintenance
• 19:35 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2190', diff saved to https://phabricator.wikimedia.org/P60146 and previous config saved to /var/cache/conftool/dbconfig/20240409-193517-arnaudb.json
• 19:35 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2197.codfw.wmnet with reason: Maintenance
• 19:35 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2197.codfw.wmnet with reason: Maintenance
• 19:35 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2189 (T360332)', diff saved to https://phabricator.wikimedia.org/P60145 and previous config saved to /var/cache/conftool/dbconfig/20240409-193507-arnaudb.json
• 19:20 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2190 (T360332)', diff saved to https://phabricator.wikimedia.org/P60143 and previous config saved to /var/cache/conftool/dbconfig/20240409-192010-arnaudb.json
• 19:20 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P60142 and previous config saved to /var/cache/conftool/dbconfig/20240409-192000-arnaudb.json
• 19:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2190 (T360332)', diff saved to https://phabricator.wikimedia.org/P60141 and previous config saved to /var/cache/conftool/dbconfig/20240409-190459-arnaudb.json
• 19:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P60140 and previous config saved to /var/cache/conftool/dbconfig/20240409-190452-arnaudb.json
• 19:05 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2190.codfw.wmnet with reason: Maintenance
• 19:04 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2190.codfw.wmnet with reason: Maintenance
• 19:04 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2177 (T360332)', diff saved to https://phabricator.wikimedia.org/P60139 and previous config saved to /var/cache/conftool/dbconfig/20240409-190436-arnaudb.json
• 18:58 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1221 (T356166)', diff saved to https://phabricator.wikimedia.org/P60138 and previous config saved to /var/cache/conftool/dbconfig/20240409-185817-marostegui.json
• 18:58 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
• 18:58 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
• 18:58 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1221.eqiad.wmnet with reason: Maintenance
• 18:57 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1221.eqiad.wmnet with reason: Maintenance
• 18:57 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1199 (T356166)', diff saved to https://phabricator.wikimedia.org/P60137 and previous config saved to /var/cache/conftool/dbconfig/20240409-185736-marostegui.json
• 18:56 eoghan@cumin1002: END (PASS) - Cookbook sre.gitlab.failover (exit_code=0) Failover of gitlab from gitlab1003.wikimedia.org to gitlab1004.wikimedia.org
• 18:54 eoghan@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) 'https://gitlab-replica.wikimedia.org/ https://gitlab-replica-old.wikimedia.org/' on all recursors
• 18:54 eoghan@cumin1002: START - Cookbook sre.dns.wipe-cache 'https://gitlab-replica.wikimedia.org/ https://gitlab-replica-old.wikimedia.org/' on all recursors
• 18:49 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2189 (T360332)', diff saved to https://phabricator.wikimedia.org/P60136 and previous config saved to /var/cache/conftool/dbconfig/20240409-184944-arnaudb.json
• 18:49 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P60135 and previous config saved to /var/cache/conftool/dbconfig/20240409-184929-arnaudb.json
• 18:47 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2189 (T360332)', diff saved to https://phabricator.wikimedia.org/P60134 and previous config saved to /var/cache/conftool/dbconfig/20240409-184702-arnaudb.json
• 18:46 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2189.codfw.wmnet with reason: Maintenance
• 18:46 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2189.codfw.wmnet with reason: Maintenance
• 18:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T360332)', diff saved to https://phabricator.wikimedia.org/P60133 and previous config saved to /var/cache/conftool/dbconfig/20240409-184640-arnaudb.json
• 18:43 ebysans@deploy1002: Finished deploy [airflow-dags/analytics@875e0d2]: (no justification provided) (duration: 00m 26s)
• 18:43 ebysans@deploy1002: Started deploy [airflow-dags/analytics@875e0d2]: (no justification provided)
• 18:42 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P60132 and previous config saved to /var/cache/conftool/dbconfig/20240409-184228-marostegui.json
• 18:41 SandraEbele_: deploying airflow dag to fix mediawiki_history_metrics_monthly dag
• 18:34 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P60131 and previous config saved to /var/cache/conftool/dbconfig/20240409-183421-arnaudb.json
• 18:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P60130 and previous config saved to /var/cache/conftool/dbconfig/20240409-183132-arnaudb.json
• 18:27 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P60129 and previous config saved to /var/cache/conftool/dbconfig/20240409-182721-marostegui.json
• 18:19 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2177 (T360332)', diff saved to https://phabricator.wikimedia.org/P60128 and previous config saved to /var/cache/conftool/dbconfig/20240409-181914-arnaudb.json
• 18:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P60127 and previous config saved to /var/cache/conftool/dbconfig/20240409-181625-arnaudb.json
• 18:12 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1199 (T356166)', diff saved to https://phabricator.wikimedia.org/P60126 and previous config saved to /var/cache/conftool/dbconfig/20240409-181213-marostegui.json
• 18:03 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2177 (T360332)', diff saved to https://phabricator.wikimedia.org/P60125 and previous config saved to /var/cache/conftool/dbconfig/20240409-180306-arnaudb.json
• 18:03 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2177.codfw.wmnet with reason: Maintenance
• 18:02 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2177.codfw.wmnet with reason: Maintenance
• 18:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2156 (T360332)', diff saved to https://phabricator.wikimedia.org/P60124 and previous config saved to /var/cache/conftool/dbconfig/20240409-180242-arnaudb.json
• 18:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T360332)', diff saved to https://phabricator.wikimedia.org/P60123 and previous config saved to /var/cache/conftool/dbconfig/20240409-180117-arnaudb.json
• 17:58 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2175 (T360332)', diff saved to https://phabricator.wikimedia.org/P60122 and previous config saved to /var/cache/conftool/dbconfig/20240409-175837-arnaudb.json
• 17:58 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2175.codfw.wmnet with reason: Maintenance
• 17:58 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2175.codfw.wmnet with reason: Maintenance
• 17:58 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2148 (T360332)', diff saved to https://phabricator.wikimedia.org/P60121 and previous config saved to /var/cache/conftool/dbconfig/20240409-175813-arnaudb.json
• 17:47 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P60120 and previous config saved to /var/cache/conftool/dbconfig/20240409-174734-arnaudb.json
• 17:43 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P60119 and previous config saved to /var/cache/conftool/dbconfig/20240409-174306-arnaudb.json
• 17:32 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P60118 and previous config saved to /var/cache/conftool/dbconfig/20240409-173226-arnaudb.json
• 17:27 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P60117 and previous config saved to /var/cache/conftool/dbconfig/20240409-172758-arnaudb.json
• 17:25 denisse: Delete unused grafana-labs TLS certificates - T360414
• 17:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2156 (T360332)', diff saved to https://phabricator.wikimedia.org/P60116 and previous config saved to /var/cache/conftool/dbconfig/20240409-171719-arnaudb.json
• 17:16 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1113.eqiad.wmnet,service=(cdn|ats-be)
• 17:15 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1113.eqiad.wmnet with OS bullseye
• 17:12 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2148 (T360332)', diff saved to https://phabricator.wikimedia.org/P60115 and previous config saved to /var/cache/conftool/dbconfig/20240409-171251-arnaudb.json
• 17:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2148 (T360332)', diff saved to https://phabricator.wikimedia.org/P60114 and previous config saved to /var/cache/conftool/dbconfig/20240409-171009-arnaudb.json
• 17:10 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2148.codfw.wmnet with reason: Maintenance
• 17:09 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2148.codfw.wmnet with reason: Maintenance
• 17:09 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2138 (T360332)', diff saved to https://phabricator.wikimedia.org/P60113 and previous config saved to /var/cache/conftool/dbconfig/20240409-170946-arnaudb.json
• 17:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2156 (T360332)', diff saved to https://phabricator.wikimedia.org/P60112 and previous config saved to /var/cache/conftool/dbconfig/20240409-170234-arnaudb.json
• 17:02 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance
• 17:02 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance
• 17:02 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2156.codfw.wmnet with reason: Maintenance
• 17:02 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2156.codfw.wmnet with reason: Maintenance
• 17:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2149 (T360332)', diff saved to https://phabricator.wikimedia.org/P60111 and previous config saved to /var/cache/conftool/dbconfig/20240409-170155-arnaudb.json
• 16:56 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1113.eqiad.wmnet with reason: host reimage
• 16:54 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2138', diff saved to https://phabricator.wikimedia.org/P60110 and previous config saved to /var/cache/conftool/dbconfig/20240409-165438-arnaudb.json
• 16:54 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1113.eqiad.wmnet with reason: host reimage
• 16:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P60109 and previous config saved to /var/cache/conftool/dbconfig/20240409-164647-arnaudb.json
• 16:39 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2138', diff saved to https://phabricator.wikimedia.org/P60108 and previous config saved to /var/cache/conftool/dbconfig/20240409-163931-arnaudb.json
• 16:38 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp1113.eqiad.wmnet with OS bullseye
• 16:37 sukhe@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp1113.eqiad.wmnet with OS bullseye
• 16:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P60107 and previous config saved to /var/cache/conftool/dbconfig/20240409-163140-arnaudb.json
• 16:24 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2138 (T360332)', diff saved to https://phabricator.wikimedia.org/P60106 and previous config saved to /var/cache/conftool/dbconfig/20240409-162423-arnaudb.json
• 16:24 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp1113.eqiad.wmnet with OS bullseye
• 16:21 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2138 (T360332)', diff saved to https://phabricator.wikimedia.org/P60105 and previous config saved to /var/cache/conftool/dbconfig/20240409-162142-arnaudb.json
• 16:21 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2138.codfw.wmnet with reason: Maintenance
• 16:21 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2138.codfw.wmnet with reason: Maintenance
• 16:21 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2126 (T360332)', diff saved to https://phabricator.wikimedia.org/P60104 and previous config saved to /var/cache/conftool/dbconfig/20240409-162119-arnaudb.json
• 16:21 sukhe@cumin1002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp1113.eqiad.wmnet
• 16:20 sukhe@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp1113.eqiad.wmnet
• 16:20 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1113.eqiad.wmnet with reason: NIC firmware upgrade and reimage
• 16:19 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1113.eqiad.wmnet with reason: NIC firmware upgrade and reimage
• 16:19 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp1113.eqiad.wmnet,service=(cdn|ats-be)
• 16:16 sukhe: depool cp1113 for PXE boot issue related testing T350179
• 16:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2149 (T360332)', diff saved to https://phabricator.wikimedia.org/P60103 and previous config saved to /var/cache/conftool/dbconfig/20240409-161632-arnaudb.json
• 16:13 denisse: Deleting unused webperf TLS certificates - T360414
• 16:13 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dumpsdata1005.eqiad.wmnet with OS bullseye
• 16:06 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp4052.ulsfo.wmnet,service=(cdn|ats-be)
• 16:06 sukhe: pool cp4052 after reimaging and new NIC firmware
• 16:06 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P60102 and previous config saved to /var/cache/conftool/dbconfig/20240409-160612-arnaudb.json
• 16:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2149 (T360332)', diff saved to https://phabricator.wikimedia.org/P60101 and previous config saved to /var/cache/conftool/dbconfig/20240409-160225-arnaudb.json
• 16:02 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2149.codfw.wmnet with reason: Maintenance
• 16:02 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2149.codfw.wmnet with reason: Maintenance
• 16:01 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host matomo1003.eqiad.wmnet with OS bookworm
• 15:56 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dumpsdata1005.eqiad.wmnet with reason: host reimage
• 15:54 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4052.ulsfo.wmnet with OS bullseye
• 15:54 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on dumpsdata1005.eqiad.wmnet with reason: host reimage
• 15:51 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P60100 and previous config saved to /var/cache/conftool/dbconfig/20240409-155104-arnaudb.json
• 15:48 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2139.codfw.wmnet with reason: Maintenance
• 15:48 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2139.codfw.wmnet with reason: Maintenance
• 15:45 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on matomo1003.eqiad.wmnet with reason: host reimage
• 15:42 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on matomo1003.eqiad.wmnet with reason: host reimage
• 15:40 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host dumpsdata1005.eqiad.wmnet with OS bullseye
• 15:39 moritzm: installing python2.7 security updates
• 15:35 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2126 (T360332)', diff saved to https://phabricator.wikimedia.org/P60099 and previous config saved to /var/cache/conftool/dbconfig/20240409-153557-arnaudb.json
• 15:35 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2109.codfw.wmnet with reason: Maintenance
• 15:35 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2109.codfw.wmnet with reason: Maintenance
• 15:35 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2105 (T360332)', diff saved to https://phabricator.wikimedia.org/P60098 and previous config saved to /var/cache/conftool/dbconfig/20240409-153512-arnaudb.json
• 15:33 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2126 (T360332)', diff saved to https://phabricator.wikimedia.org/P60097 and previous config saved to /var/cache/conftool/dbconfig/20240409-153315-arnaudb.json
• 15:33 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance
• 15:33 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance
• 15:33 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2126.codfw.wmnet with reason: Maintenance
• 15:33 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2126.codfw.wmnet with reason: Maintenance
• 15:33 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2125 (T360332)', diff saved to https://phabricator.wikimedia.org/P60096 and previous config saved to /var/cache/conftool/dbconfig/20240409-153257-arnaudb.json
• 15:32 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage
• 15:32 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host matomo1003.eqiad.wmnet with OS bookworm
• 15:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-ulsfo and not P{cp[4037,4041,4045,4049,4052].ulsfo.wmnet} and A:cp
• 15:26 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage
• 15:24 btullis@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dumpsdata1005.eqiad.wmnet with OS bullseye
• 15:22 hnowlan@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
• 15:22 hnowlan@deploy1002: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
• 15:20 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
• 15:20 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
• 15:20 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P60095 and previous config saved to /var/cache/conftool/dbconfig/20240409-152005-arnaudb.json
• 15:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2125', diff saved to https://phabricator.wikimedia.org/P60094 and previous config saved to /var/cache/conftool/dbconfig/20240409-151750-arnaudb.json
• 15:16 eoghan@cumin1002: START - Cookbook sre.gitlab.failover Failover of gitlab from gitlab1003.wikimedia.org to gitlab1004.wikimedia.org
• 15:06 isaranto@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
• 15:04 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P60093 and previous config saved to /var/cache/conftool/dbconfig/20240409-150458-arnaudb.json
• 15:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2125', diff saved to https://phabricator.wikimedia.org/P60092 and previous config saved to /var/cache/conftool/dbconfig/20240409-150242-arnaudb.json
• 15:02 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp4052.ulsfo.wmnet with OS bullseye
• 15:01 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-ulsfo and not P{cp[4037,4041,4045,4049,4052].ulsfo.wmnet} and A:cp
• 14:49 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2105 (T360332)', diff saved to https://phabricator.wikimedia.org/P60091 and previous config saved to /var/cache/conftool/dbconfig/20240409-144950-arnaudb.json
• 14:47 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2125 (T360332)', diff saved to https://phabricator.wikimedia.org/P60090 and previous config saved to /var/cache/conftool/dbconfig/20240409-144735-arnaudb.json
• 14:44 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2125 (T360332)', diff saved to https://phabricator.wikimedia.org/P60089 and previous config saved to /var/cache/conftool/dbconfig/20240409-144447-arnaudb.json
• 14:44 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2125.codfw.wmnet with reason: Maintenance
• 14:44 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2125.codfw.wmnet with reason: Maintenance
• 14:44 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2107 (T360332)', diff saved to https://phabricator.wikimedia.org/P60088 and previous config saved to /var/cache/conftool/dbconfig/20240409-144424-arnaudb.json
• 14:35 sukhe@cumin1002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp4052.ulsfo.wmnet
• 14:34 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2105 (T360332)', diff saved to https://phabricator.wikimedia.org/P60087 and previous config saved to /var/cache/conftool/dbconfig/20240409-143445-arnaudb.json
• 14:34 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2105.codfw.wmnet with reason: Maintenance
• 14:34 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2105.codfw.wmnet with reason: Maintenance
• 14:33 sukhe@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp4052.ulsfo.wmnet
• 14:33 btullis@cumin1002: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host matomo1003.eqiad.wmnet
• 14:33 btullis@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host matomo1003.eqiad.wmnet with OS bookworm
• 14:30 sukhe@cumin1002: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts cp4052.ulsfo.wmnet
• 14:30 sukhe@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp4052.ulsfo.wmnet
• 14:29 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
• 14:29 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
• 14:29 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2107', diff saved to https://phabricator.wikimedia.org/P60086 and previous config saved to /var/cache/conftool/dbconfig/20240409-142916-arnaudb.json
• 14:28 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4052.ulsfo.wmnet with OS bullseye
• 14:24 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1240.eqiad.wmnet with reason: Maintenance
• 14:24 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1240.eqiad.wmnet with reason: Maintenance
• 14:24 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1223 (T360332)', diff saved to https://phabricator.wikimedia.org/P60085 and previous config saved to /var/cache/conftool/dbconfig/20240409-142414-arnaudb.json
• 14:18 elukey@cumin1002: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:aqs-eqiad: Deploy new Truststore - elukey@cumin1002
• 14:14 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2107', diff saved to https://phabricator.wikimedia.org/P60084 and previous config saved to /var/cache/conftool/dbconfig/20240409-141410-arnaudb.json
• 14:09 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P60083 and previous config saved to /var/cache/conftool/dbconfig/20240409-140906-arnaudb.json
• 14:07 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage
• 14:07 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host dumpsdata1005.eqiad.wmnet with OS bullseye
• 14:04 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage
• 14:03 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
• 14:02 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
• 13:59 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2107 (T360332)', diff saved to https://phabricator.wikimedia.org/P60082 and previous config saved to /var/cache/conftool/dbconfig/20240409-135902-arnaudb.json
• 13:57 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dumpsdata1004.eqiad.wmnet with OS bullseye
• 13:57 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) Will create a clone of db2112.codfw.wmnet onto db2212.codfw.wmnet
• 13:56 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2107 (T360332)', diff saved to https://phabricator.wikimedia.org/P60081 and previous config saved to /var/cache/conftool/dbconfig/20240409-135621-arnaudb.json
• 13:56 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2107.codfw.wmnet with reason: Maintenance
• 13:56 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2107.codfw.wmnet with reason: Maintenance
• 13:55 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2097.codfw.wmnet with reason: Maintenance
• 13:55 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2097.codfw.wmnet with reason: Maintenance
• 13:54 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
• 13:54 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
• 13:54 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
• 13:54 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1246.eqiad.wmnet with reason: Maintenance
• 13:54 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1246.eqiad.wmnet with reason: Maintenance
• 13:54 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
• 13:53 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P60080 and previous config saved to /var/cache/conftool/dbconfig/20240409-135359-arnaudb.json
• 13:53 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1239.eqiad.wmnet with reason: Maintenance
• 13:53 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1239.eqiad.wmnet with reason: Maintenance
• 13:53 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1233 (T360332)', diff saved to https://phabricator.wikimedia.org/P60079 and previous config saved to /var/cache/conftool/dbconfig/20240409-135335-arnaudb.json
• 13:42 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp4052.ulsfo.wmnet with OS bullseye
• 13:42 sukhe@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4052.ulsfo.wmnet with OS bullseye
• 13:42 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dumpsdata1004.eqiad.wmnet with reason: host reimage
• 13:38 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on dumpsdata1004.eqiad.wmnet with reason: host reimage
• 13:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1223 (T360332)', diff saved to https://phabricator.wikimedia.org/P60078 and previous config saved to /var/cache/conftool/dbconfig/20240409-133852-arnaudb.json
• 13:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P60077 and previous config saved to /var/cache/conftool/dbconfig/20240409-133827-arnaudb.json
• 13:35 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp4052.ulsfo.wmnet with OS bullseye
• 13:33 sukhe@cumin1002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp4052.ulsfo.wmnet
• 13:33 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp4052.ulsfo.wmnet
• 13:32 jforrester@deploy1002: Finished scap: Backport for zhwiki:Add centralauth-createlocal to ipblock exempt granter (T361184) (duration: 14m 31s)
• 13:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1223 (T360332)', diff saved to https://phabricator.wikimedia.org/P60076 and previous config saved to /var/cache/conftool/dbconfig/20240409-133135-arnaudb.json
• 13:31 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1223.eqiad.wmnet with reason: Maintenance
• 13:31 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1223.eqiad.wmnet with reason: Maintenance
• 13:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1212 (T360332)', diff saved to https://phabricator.wikimedia.org/P60075 and previous config saved to /var/cache/conftool/dbconfig/20240409-133112-arnaudb.json
• 13:25 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host dumpsdata1004.eqiad.wmnet with OS bullseye
• 13:23 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P60074 and previous config saved to /var/cache/conftool/dbconfig/20240409-132320-arnaudb.json
• 13:21 jforrester@deploy1002: sdhehua and jforrester: Continuing with sync
• 13:21 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host matomo1003.eqiad.wmnet with OS bookworm
• 13:20 jforrester@deploy1002: sdhehua and jforrester: Backport for zhwiki:Add centralauth-createlocal to ipblock exempt granter (T361184) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 13:18 sukhe@cumin1002: START - Cookbook sre.hosts.reboot-single for host cp4052.ulsfo.wmnet
• 13:17 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4052.ulsfo.wmnet with reason: BIOS firmware upgrade
• 13:17 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp4052.ulsfo.wmnet with reason: BIOS firmware upgrade
• 13:17 jforrester@deploy1002: Started scap: Backport for zhwiki:Add centralauth-createlocal to ipblock exempt granter (T361184)
• 13:16 sukhe@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp4052.ulsfo.wmnet
• 13:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P60073 and previous config saved to /var/cache/conftool/dbconfig/20240409-131605-arnaudb.json
• 13:16 sukhe: depool cp4052 for firmware upgrade
• 13:16 jforrester@deploy1002: Finished scap: Backport for ExtensionDistributor: Add REL1_42 as a beta (T359844) (duration: 14m 09s)
• 13:14 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on stat1010.eqiad.wmnet with reason: Connecting GPU power cable
• 13:14 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on stat1010.eqiad.wmnet with reason: Connecting GPU power cable
• 13:08 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1233 (T360332)', diff saved to https://phabricator.wikimedia.org/P60072 and previous config saved to /var/cache/conftool/dbconfig/20240409-130812-arnaudb.json
• 13:06 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1233 (T360332)', diff saved to https://phabricator.wikimedia.org/P60071 and previous config saved to /var/cache/conftool/dbconfig/20240409-130543-arnaudb.json
• 13:05 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1233.eqiad.wmnet with reason: Maintenance
• 13:05 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1233.eqiad.wmnet with reason: Maintenance
• 13:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1229 (T360332)', diff saved to https://phabricator.wikimedia.org/P60070 and previous config saved to /var/cache/conftool/dbconfig/20240409-130520-arnaudb.json
• 13:04 jforrester@deploy1002: jforrester: Continuing with sync
• 13:04 jforrester@deploy1002: jforrester: Backport for ExtensionDistributor: Add REL1_42 as a beta (T359844) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 13:01 jforrester@deploy1002: Started scap: Backport for ExtensionDistributor: Add REL1_42 as a beta (T359844)
• 13:00 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P60069 and previous config saved to /var/cache/conftool/dbconfig/20240409-130057-arnaudb.json
• 12:53 vgutierrez: uploaded golang-github-gopacket-gopacket_1.2.0-2~wmf1 to apt.wm.o (bookworm)
• 12:52 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-role (exit_code=0) for role: wmcs::openstack::codfw1dev::cinder_backups
• 12:50 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
• 12:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P60068 and previous config saved to /var/cache/conftool/dbconfig/20240409-125012-arnaudb.json
• 12:45 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1212 (T360332)', diff saved to https://phabricator.wikimedia.org/P60067 and previous config saved to /var/cache/conftool/dbconfig/20240409-124550-arnaudb.json
• 12:45 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
• 12:44 elukey@cumin1002: START - Cookbook sre.cassandra.roll-restart for nodes matching A:aqs-eqiad: Deploy new Truststore - elukey@cumin1002
• 12:43 jmm@cumin2002: START - Cookbook sre.puppet.migrate-role for role: wmcs::openstack::codfw1dev::cinder_backups
• 12:25 gmodena@deploy1002: Finished deploy [analytics/refinery@d45a15b]: Regular analytics weekly train [analytics/refinery@d45a15b6] (duration: 15m 41s)
• 12:21 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host cloudbackup1001-dev.eqiad.wmnet
• 12:20 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P60063 and previous config saved to /var/cache/conftool/dbconfig/20240409-122050-arnaudb.json
• 12:19 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1229 (T360332)', diff saved to https://phabricator.wikimedia.org/P60062 and previous config saved to /var/cache/conftool/dbconfig/20240409-121958-arnaudb.json
• 12:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1229 (T360332)', diff saved to https://phabricator.wikimedia.org/P60061 and previous config saved to /var/cache/conftool/dbconfig/20240409-121722-arnaudb.json
• 12:17 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1229.eqiad.wmnet with reason: Maintenance
• 12:17 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1229.eqiad.wmnet with reason: Maintenance
• 12:16 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1225.eqiad.wmnet with reason: Maintenance
• 12:16 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1225.eqiad.wmnet with reason: Maintenance
• 12:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1197 (T360332)', diff saved to https://phabricator.wikimedia.org/P60060 and previous config saved to /var/cache/conftool/dbconfig/20240409-121622-arnaudb.json
• 12:09 gmodena@deploy1002: Started deploy [analytics/refinery@d45a15b]: Regular analytics weekly train [analytics/refinery@d45a15b6]
• 12:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P60059 and previous config saved to /var/cache/conftool/dbconfig/20240409-120542-arnaudb.json
• 12:02 btullis@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dumpsdata1004.eqiad.wmnet with OS bullseye
• 12:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P60058 and previous config saved to /var/cache/conftool/dbconfig/20240409-120115-arnaudb.json
• 11:56 btullis@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM matomo1003.eqiad.wmnet - btullis@cumin1002"
• 11:55 btullis@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM matomo1003.eqiad.wmnet - btullis@cumin1002"
• 11:54 btullis@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) matomo1003.eqiad.wmnet on all recursors
• 11:54 btullis@cumin1002: START - Cookbook sre.dns.wipe-cache matomo1003.eqiad.wmnet on all recursors
• 11:54 btullis@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 11:54 btullis@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM matomo1003.eqiad.wmnet - btullis@cumin1002"
• 11:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T360332)', diff saved to https://phabricator.wikimedia.org/P60057 and previous config saved to /var/cache/conftool/dbconfig/20240409-115035-arnaudb.json
• 11:49 hnowlan@deploy1002: helmfile [codfw] [main] DONE helmfile.d/services/mw-jobrunner : sync
• 11:48 hnowlan@deploy1002: helmfile [codfw] [canary] DONE helmfile.d/services/mw-jobrunner : sync
• 11:47 hnowlan@deploy1002: helmfile [codfw] [canary] START helmfile.d/services/mw-jobrunner : sync
• 11:47 hnowlan@deploy1002: helmfile [codfw] [main] START helmfile.d/services/mw-jobrunner : sync
• 11:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P60056 and previous config saved to /var/cache/conftool/dbconfig/20240409-114607-arnaudb.json
• 11:44 arnaudb@cumin1002: START - Cookbook sre.mysql.clone Will create a clone of db2112.codfw.wmnet onto db2212.codfw.wmnet
• 11:44 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1198 (T360332)', diff saved to https://phabricator.wikimedia.org/P60055 and previous config saved to /var/cache/conftool/dbconfig/20240409-114411-arnaudb.json
• 11:44 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1198.eqiad.wmnet with reason: Maintenance
• 11:43 hnowlan@deploy1002: helmfile [eqiad] [main] DONE helmfile.d/services/mw-jobrunner : sync
• 11:43 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1198.eqiad.wmnet with reason: Maintenance
• 11:43 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T360332)', diff saved to https://phabricator.wikimedia.org/P60054 and previous config saved to /var/cache/conftool/dbconfig/20240409-114349-arnaudb.json
• 11:43 arnaudb@cumin1002: dbctl commit (dc=all): 'Cloning db2112 in db2212 for T355422', diff saved to https://phabricator.wikimedia.org/P60053 and previous config saved to /var/cache/conftool/dbconfig/20240409-114302-arnaudb.json
• 11:42 hnowlan@deploy1002: helmfile [eqiad] [canary] DONE helmfile.d/services/mw-jobrunner : sync
• 11:42 hnowlan@deploy1002: helmfile [eqiad] [main] START helmfile.d/services/mw-jobrunner : sync
• 11:42 hnowlan@deploy1002: helmfile [eqiad] [canary] START helmfile.d/services/mw-jobrunner : sync
• 11:41 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2212.codfw.wmnet with reason: provisionning db2212.codfw.wmnet - T355422
• 11:41 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2212.codfw.wmnet with reason: provisionning db2212.codfw.wmnet - T355422
• 11:41 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2112.codfw.wmnet with reason: provisionning db2212.codfw.wmnet - T355422
• 11:40 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2112.codfw.wmnet with reason: provisionning db2212.codfw.wmnet - T355422
• 11:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1197 (T360332)', diff saved to https://phabricator.wikimedia.org/P60051 and previous config saved to /var/cache/conftool/dbconfig/20240409-113100-arnaudb.json
• 11:29 btullis@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM matomo1003.eqiad.wmnet - btullis@cumin1002"
• 11:28 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P60050 and previous config saved to /var/cache/conftool/dbconfig/20240409-112841-arnaudb.json
• 11:27 btullis@cumin1002: START - Cookbook sre.dns.netbox
• 11:27 btullis@cumin1002: START - Cookbook sre.ganeti.makevm for new host matomo1003.eqiad.wmnet
• 11:27 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1197 (T360332)', diff saved to https://phabricator.wikimedia.org/P60049 and previous config saved to /var/cache/conftool/dbconfig/20240409-112728-arnaudb.json
• 11:27 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1197.eqiad.wmnet with reason: Maintenance
• 11:27 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1197.eqiad.wmnet with reason: Maintenance
• 11:27 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1188 (T360332)', diff saved to https://phabricator.wikimedia.org/P60048 and previous config saved to /var/cache/conftool/dbconfig/20240409-112705-arnaudb.json
• 11:15 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host dumpsdata1004.eqiad.wmnet with OS bullseye
• 11:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P60047 and previous config saved to /var/cache/conftool/dbconfig/20240409-111334-arnaudb.json
• 11:11 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P60046 and previous config saved to /var/cache/conftool/dbconfig/20240409-111157-arnaudb.json
• 11:01 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host snapshot1015.eqiad.wmnet with OS bullseye
• 10:58 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T360332)', diff saved to https://phabricator.wikimedia.org/P60045 and previous config saved to /var/cache/conftool/dbconfig/20240409-105827-arnaudb.json
• 10:58 marostegui@cumin1002: dbctl commit (dc=all): 'db1162 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P60044 and previous config saved to /var/cache/conftool/dbconfig/20240409-105814-root.json
• 10:56 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P60043 and previous config saved to /var/cache/conftool/dbconfig/20240409-105650-arnaudb.json
• 10:48 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1189 (T360332)', diff saved to https://phabricator.wikimedia.org/P60042 and previous config saved to /var/cache/conftool/dbconfig/20240409-104853-arnaudb.json
• 10:48 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1189.eqiad.wmnet with reason: Maintenance
• 10:48 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1189.eqiad.wmnet with reason: Maintenance
• 10:48 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T360332)', diff saved to https://phabricator.wikimedia.org/P60041 and previous config saved to /var/cache/conftool/dbconfig/20240409-104830-arnaudb.json
• 10:45 jayme@deploy1002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
• 10:45 jayme@deploy1002: helmfile [eqiad] START helmfile.d/admin 'apply'.
• 10:44 jayme@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
• 10:43 marostegui@cumin1002: dbctl commit (dc=all): 'db1162 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P60040 and previous config saved to /var/cache/conftool/dbconfig/20240409-104308-root.json
• 10:42 jayme@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
• 10:41 jayme@deploy1002: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
• 10:41 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1188 (T360332)', diff saved to https://phabricator.wikimedia.org/P60039 and previous config saved to /var/cache/conftool/dbconfig/20240409-104143-arnaudb.json
• 10:41 jayme@deploy1002: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
• 10:39 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1188 (T360332)', diff saved to https://phabricator.wikimedia.org/P60038 and previous config saved to /var/cache/conftool/dbconfig/20240409-103908-arnaudb.json
• 10:39 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1188.eqiad.wmnet with reason: Maintenance
• 10:38 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1188.eqiad.wmnet with reason: Maintenance
• 10:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T360332)', diff saved to https://phabricator.wikimedia.org/P60037 and previous config saved to /var/cache/conftool/dbconfig/20240409-103845-arnaudb.json
• 10:34 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on snapshot1015.eqiad.wmnet with reason: host reimage
• 10:33 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P60036 and previous config saved to /var/cache/conftool/dbconfig/20240409-103323-arnaudb.json
• 10:32 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on db2097.codfw.wmnet with reason: host weirdness and possible decom
• 10:31 jynus@cumin1002: START - Cookbook sre.hosts.downtime for 14 days, 0:00:00 on db2097.codfw.wmnet with reason: host weirdness and possible decom
• 10:31 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on snapshot1015.eqiad.wmnet with reason: host reimage
• 10:28 marostegui@cumin1002: dbctl commit (dc=all): 'db1162 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P60035 and previous config saved to /var/cache/conftool/dbconfig/20240409-102803-root.json
• 10:23 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P60034 and previous config saved to /var/cache/conftool/dbconfig/20240409-102337-arnaudb.json
• 10:23 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 (re)pooling @ 100%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P60033 and previous config saved to /var/cache/conftool/dbconfig/20240409-102312-arnaudb.json
• 10:19 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host snapshot1015.eqiad.wmnet with OS bullseye
• 10:18 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P60032 and previous config saved to /var/cache/conftool/dbconfig/20240409-101815-arnaudb.json
• 10:12 marostegui@cumin1002: dbctl commit (dc=all): 'db1162 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P60031 and previous config saved to /var/cache/conftool/dbconfig/20240409-101257-root.json
• 10:08 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P60030 and previous config saved to /var/cache/conftool/dbconfig/20240409-100830-arnaudb.json
• 10:08 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 (re)pooling @ 75%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P60029 and previous config saved to /var/cache/conftool/dbconfig/20240409-100806-arnaudb.json
• 10:06 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1199 (T356166)', diff saved to https://phabricator.wikimedia.org/P60028 and previous config saved to /var/cache/conftool/dbconfig/20240409-100635-marostegui.json
• 10:06 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1199.eqiad.wmnet with reason: Maintenance
• 10:06 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1199.eqiad.wmnet with reason: Maintenance
• 10:06 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1190 (T356166)', diff saved to https://phabricator.wikimedia.org/P60027 and previous config saved to /var/cache/conftool/dbconfig/20240409-100612-marostegui.json
• 10:03 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T360332)', diff saved to https://phabricator.wikimedia.org/P60026 and previous config saved to /var/cache/conftool/dbconfig/20240409-100308-arnaudb.json
• 10:02 Emperor: puppet cert clean swift_eqiad T361844
• 09:57 marostegui@cumin1002: dbctl commit (dc=all): 'db1162 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P60025 and previous config saved to /var/cache/conftool/dbconfig/20240409-095751-root.json
• 09:56 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1175 (T360332)', diff saved to https://phabricator.wikimedia.org/P60024 and previous config saved to /var/cache/conftool/dbconfig/20240409-095642-arnaudb.json
• 09:56 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1175.eqiad.wmnet with reason: Maintenance
• 09:56 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1175.eqiad.wmnet with reason: Maintenance
• 09:56 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T360332)', diff saved to https://phabricator.wikimedia.org/P60023 and previous config saved to /var/cache/conftool/dbconfig/20240409-095620-arnaudb.json
• 09:53 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T360332)', diff saved to https://phabricator.wikimedia.org/P60022 and previous config saved to /var/cache/conftool/dbconfig/20240409-095323-arnaudb.json
• 09:53 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 (re)pooling @ 50%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P60021 and previous config saved to /var/cache/conftool/dbconfig/20240409-095300-arnaudb.json
• 09:51 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P60020 and previous config saved to /var/cache/conftool/dbconfig/20240409-095105-marostegui.json
• 09:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1182 (T360332)', diff saved to https://phabricator.wikimedia.org/P60019 and previous config saved to /var/cache/conftool/dbconfig/20240409-095054-arnaudb.json
• 09:50 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1182.eqiad.wmnet with reason: Maintenance
• 09:50 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1182.eqiad.wmnet with reason: Maintenance
• 09:50 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1162.eqiad.wmnet with reason: Maintenance
• 09:50 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1162.eqiad.wmnet with reason: Maintenance
• 09:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T360332)', diff saved to https://phabricator.wikimedia.org/P60018 and previous config saved to /var/cache/conftool/dbconfig/20240409-095004-arnaudb.json
• 09:46 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 09:45 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 09:42 marostegui@cumin1002: dbctl commit (dc=all): 'db1162 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P60017 and previous config saved to /var/cache/conftool/dbconfig/20240409-094246-root.json
• 09:41 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P60016 and previous config saved to /var/cache/conftool/dbconfig/20240409-094113-arnaudb.json
• 09:40 hashar@deploy1002: rebuilt and synchronized wikiversions files: group0 wikis to 1.42.0-wmf.26 refs T360158
• 09:37 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 (re)pooling @ 25%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P60015 and previous config saved to /var/cache/conftool/dbconfig/20240409-093755-arnaudb.json
• 09:35 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P60014 and previous config saved to /var/cache/conftool/dbconfig/20240409-093551-marostegui.json
• 09:34 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P60013 and previous config saved to /var/cache/conftool/dbconfig/20240409-093457-arnaudb.json
• 09:27 marostegui@cumin1002: dbctl commit (dc=all): 'db1162 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P60012 and previous config saved to /var/cache/conftool/dbconfig/20240409-092740-root.json
• 09:26 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P60011 and previous config saved to /var/cache/conftool/dbconfig/20240409-092605-arnaudb.json
• 09:22 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 (re)pooling @ 16%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P60010 and previous config saved to /var/cache/conftool/dbconfig/20240409-092249-arnaudb.json
• 09:20 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1190 (T356166)', diff saved to https://phabricator.wikimedia.org/P60009 and previous config saved to /var/cache/conftool/dbconfig/20240409-092043-marostegui.json
• 09:19 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P60008 and previous config saved to /var/cache/conftool/dbconfig/20240409-091949-arnaudb.json
• 09:18 hashar@deploy1002: Finished scap: testwikis wikis to 1.42.0-wmf.26 refs T360158 (duration: 50m 11s)
• 09:11 moritzm: installing postgresql-13 security updates
• 09:11 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T360332)', diff saved to https://phabricator.wikimedia.org/P60007 and previous config saved to /var/cache/conftool/dbconfig/20240409-091057-arnaudb.json
• 09:07 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 (re)pooling @ 8%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P60006 and previous config saved to /var/cache/conftool/dbconfig/20240409-090744-arnaudb.json
• 09:04 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T360332)', diff saved to https://phabricator.wikimedia.org/P60005 and previous config saved to /var/cache/conftool/dbconfig/20240409-090442-arnaudb.json
• 09:04 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1166 (T360332)', diff saved to https://phabricator.wikimedia.org/P60004 and previous config saved to /var/cache/conftool/dbconfig/20240409-090435-arnaudb.json
• 09:04 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1166.eqiad.wmnet with reason: Maintenance
• 09:04 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1166.eqiad.wmnet with reason: Maintenance
• 09:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1156 (T360332)', diff saved to https://phabricator.wikimedia.org/P60003 and previous config saved to /var/cache/conftool/dbconfig/20240409-090210-arnaudb.json
• 09:02 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
• 09:01 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
• 09:01 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1156.eqiad.wmnet with reason: Maintenance
• 09:01 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1156.eqiad.wmnet with reason: Maintenance
• 08:59 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1150.eqiad.wmnet with reason: Maintenance
• 08:59 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1150.eqiad.wmnet with reason: Maintenance
• 08:56 marostegui@cumin1002: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1162.eqiad.wmnet onto db1246.eqiad.wmnet
• 08:52 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 (re)pooling @ 4%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P60002 and previous config saved to /var/cache/conftool/dbconfig/20240409-085238-arnaudb.json
• 08:37 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 (re)pooling @ 2%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P60001 and previous config saved to /var/cache/conftool/dbconfig/20240409-083733-arnaudb.json
• 08:28 hashar@deploy1002: Started scap: testwikis wikis to 1.42.0-wmf.26 refs T360158
• 08:22 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 (re)pooling @ 1%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P60000 and previous config saved to /var/cache/conftool/dbconfig/20240409-082227-arnaudb.json
• 08:17 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2113.codfw.wmnet with OS bookworm
• 07:56 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2113.codfw.wmnet with reason: host reimage
• 07:54 Emperor: puppet cert clean swift_codfw T361844
• 07:53 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2113.codfw.wmnet with reason: host reimage
• 07:37 arnaudb@cumin1002: START - Cookbook sre.hosts.reimage for host db2113.codfw.wmnet with OS bookworm
• 07:34 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 depool for reimage T360116', diff saved to https://phabricator.wikimedia.org/P59999 and previous config saved to /var/cache/conftool/dbconfig/20240409-073406-arnaudb.json
• 07:33 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2213.codfw.wmnet with reason: Silence for reimage
• 07:33 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2213.codfw.wmnet with reason: Silence for reimage
• 07:29 kartik@deploy1002: Finished scap: Backport for ContentTranslation: Limit publishing in zhwiki for extendedconfirmed users only (T349959) (duration: 25m 30s)
• 07:16 kartik@deploy1002: kartik: Continuing with sync
• 07:09 kartik@deploy1002: kartik: Backport for ContentTranslation: Limit publishing in zhwiki for extendedconfirmed users only (T349959) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 07:08 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1150.eqiad.wmnet with reason: Maintenance
• 07:08 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1150.eqiad.wmnet with reason: Maintenance
• 07:06 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2219 (T360332)', diff saved to https://phabricator.wikimedia.org/P59998 and previous config saved to /var/cache/conftool/dbconfig/20240409-070613-arnaudb.json
• 07:04 kartik@deploy1002: Started scap: Backport for ContentTranslation: Limit publishing in zhwiki for extendedconfirmed users only (T349959)
• 07:01 marostegui@cumin1002: START - Cookbook sre.mysql.clone of db1162.eqiad.wmnet onto db1246.eqiad.wmnet
• 06:51 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2219', diff saved to https://phabricator.wikimedia.org/P59997 and previous config saved to /var/cache/conftool/dbconfig/20240409-065105-arnaudb.json
• 06:35 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2219', diff saved to https://phabricator.wikimedia.org/P59996 and previous config saved to /var/cache/conftool/dbconfig/20240409-063558-arnaudb.json
• 06:20 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2219 (T360332)', diff saved to https://phabricator.wikimedia.org/P59995 and previous config saved to /var/cache/conftool/dbconfig/20240409-062050-arnaudb.json
• 06:18 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2219 (T360332)', diff saved to https://phabricator.wikimedia.org/P59994 and previous config saved to /var/cache/conftool/dbconfig/20240409-061830-arnaudb.json
• 06:18 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2219.codfw.wmnet with reason: Maintenance
• 06:18 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2219.codfw.wmnet with reason: Maintenance
• 06:18 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2210 (T360332)', diff saved to https://phabricator.wikimedia.org/P59993 and previous config saved to /var/cache/conftool/dbconfig/20240409-061807-arnaudb.json
• 06:15 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1246.eqiad.wmnet with OS bookworm
• 06:09 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1162.eqiad.wmnet with OS bookworm
• 06:03 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2210', diff saved to https://phabricator.wikimedia.org/P59992 and previous config saved to /var/cache/conftool/dbconfig/20240409-060300-arnaudb.json
• 05:55 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1246.eqiad.wmnet with reason: host reimage
• 05:51 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1246.eqiad.wmnet with reason: host reimage
• 05:48 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1162.eqiad.wmnet with reason: host reimage
• 05:47 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2210', diff saved to https://phabricator.wikimedia.org/P59991 and previous config saved to /var/cache/conftool/dbconfig/20240409-054752-arnaudb.json
• 05:45 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1162.eqiad.wmnet with reason: host reimage
• 05:39 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db1246.eqiad.wmnet with OS bookworm
• 05:33 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db1162.eqiad.wmnet with OS bookworm
• 05:32 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2210 (T360332)', diff saved to https://phabricator.wikimedia.org/P59990 and previous config saved to /var/cache/conftool/dbconfig/20240409-053245-arnaudb.json
• 05:30 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2210 (T360332)', diff saved to https://phabricator.wikimedia.org/P59989 and previous config saved to /var/cache/conftool/dbconfig/20240409-053024-arnaudb.json
• 05:30 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2210.codfw.wmnet with reason: Maintenance
• 05:30 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1162 T362036', diff saved to https://phabricator.wikimedia.org/P59988 and previous config saved to /var/cache/conftool/dbconfig/20240409-053005-root.json
• 05:30 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2210.codfw.wmnet with reason: Maintenance
• 05:30 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2206 (T360332)', diff saved to https://phabricator.wikimedia.org/P59987 and previous config saved to /var/cache/conftool/dbconfig/20240409-053001-arnaudb.json
• 05:28 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db1222 to s2 primary and set section read-write T362036', diff saved to https://phabricator.wikimedia.org/P59986 and previous config saved to /var/cache/conftool/dbconfig/20240409-052855-marostegui.json
• 05:28 marostegui@cumin1002: dbctl commit (dc=all): 'Set s2 eqiad as read-only for maintenance - T362036', diff saved to https://phabricator.wikimedia.org/P59985 and previous config saved to /var/cache/conftool/dbconfig/20240409-052827-marostegui.json
• 05:28 marostegui: Starting s2 eqiad failover from db1162 to db1222 - T362036
• 05:14 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2206', diff saved to https://phabricator.wikimedia.org/P59984 and previous config saved to /var/cache/conftool/dbconfig/20240409-051454-arnaudb.json
• 05:10 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1222 with weight 0 T362036', diff saved to https://phabricator.wikimedia.org/P59983 and previous config saved to /var/cache/conftool/dbconfig/20240409-051027-marostegui.json
• 05:10 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 28 hosts with reason: Primary switchover s2 T362036
• 05:09 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on 28 hosts with reason: Primary switchover s2 T362036
• 04:59 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2206', diff saved to https://phabricator.wikimedia.org/P59982 and previous config saved to /var/cache/conftool/dbconfig/20240409-045946-arnaudb.json
• 04:44 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2206 (T360332)', diff saved to https://phabricator.wikimedia.org/P59981 and previous config saved to /var/cache/conftool/dbconfig/20240409-044438-arnaudb.json
• 04:42 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2206 (T360332)', diff saved to https://phabricator.wikimedia.org/P59980 and previous config saved to /var/cache/conftool/dbconfig/20240409-044216-arnaudb.json
• 04:42 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2206.codfw.wmnet with reason: Maintenance
• 04:42 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2206.codfw.wmnet with reason: Maintenance
• 04:42 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2172 (T360332)', diff saved to https://phabricator.wikimedia.org/P59979 and previous config saved to /var/cache/conftool/dbconfig/20240409-044204-arnaudb.json
• 04:26 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P59978 and previous config saved to /var/cache/conftool/dbconfig/20240409-042657-arnaudb.json
• 04:11 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P59977 and previous config saved to /var/cache/conftool/dbconfig/20240409-041149-arnaudb.json
• 03:56 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2172 (T360332)', diff saved to https://phabricator.wikimedia.org/P59976 and previous config saved to /var/cache/conftool/dbconfig/20240409-035641-arnaudb.json
• 03:54 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2172 (T360332)', diff saved to https://phabricator.wikimedia.org/P59975 and previous config saved to /var/cache/conftool/dbconfig/20240409-035422-arnaudb.json
• 03:54 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2172.codfw.wmnet with reason: Maintenance
• 03:54 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2172.codfw.wmnet with reason: Maintenance
• 03:54 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2155 (T360332)', diff saved to https://phabricator.wikimedia.org/P59974 and previous config saved to /var/cache/conftool/dbconfig/20240409-035359-arnaudb.json
• 03:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P59973 and previous config saved to /var/cache/conftool/dbconfig/20240409-033851-arnaudb.json
• 03:23 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P59972 and previous config saved to /var/cache/conftool/dbconfig/20240409-032344-arnaudb.json
• 03:08 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2155 (T360332)', diff saved to https://phabricator.wikimedia.org/P59971 and previous config saved to /var/cache/conftool/dbconfig/20240409-030836-arnaudb.json
• 03:06 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2155 (T360332)', diff saved to https://phabricator.wikimedia.org/P59970 and previous config saved to /var/cache/conftool/dbconfig/20240409-030617-arnaudb.json
• 03:06 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance
• 03:05 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance
• 03:05 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2155.codfw.wmnet with reason: Maintenance
• 03:05 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2155.codfw.wmnet with reason: Maintenance
• 03:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2147 (T360332)', diff saved to https://phabricator.wikimedia.org/P59969 and previous config saved to /var/cache/conftool/dbconfig/20240409-030537-arnaudb.json
• 02:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P59968 and previous config saved to /var/cache/conftool/dbconfig/20240409-025030-arnaudb.json
• 02:35 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P59967 and previous config saved to /var/cache/conftool/dbconfig/20240409-023522-arnaudb.json
• 02:20 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2147 (T360332)', diff saved to https://phabricator.wikimedia.org/P59966 and previous config saved to /var/cache/conftool/dbconfig/20240409-022015-arnaudb.json
• 02:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2147 (T360332)', diff saved to https://phabricator.wikimedia.org/P59965 and previous config saved to /var/cache/conftool/dbconfig/20240409-021755-arnaudb.json
• 02:17 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2147.codfw.wmnet with reason: Maintenance
• 02:17 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2147.codfw.wmnet with reason: Maintenance
• 02:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2140 (T360332)', diff saved to https://phabricator.wikimedia.org/P59964 and previous config saved to /var/cache/conftool/dbconfig/20240409-021731-arnaudb.json
• 02:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2140', diff saved to https://phabricator.wikimedia.org/P59963 and previous config saved to /var/cache/conftool/dbconfig/20240409-020223-arnaudb.json
• 01:49 ebysans@deploy1002: Finished deploy [airflow-dags/analytics@6bb821b]: (no justification provided) (duration: 00m 31s)
• 01:49 ebysans@deploy1002: Started deploy [airflow-dags/analytics@6bb821b]: (no justification provided)
• 01:47 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2140', diff saved to https://phabricator.wikimedia.org/P59962 and previous config saved to /var/cache/conftool/dbconfig/20240409-014716-arnaudb.json
• 01:32 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1190 (T356166)', diff saved to https://phabricator.wikimedia.org/P59961 and previous config saved to /var/cache/conftool/dbconfig/20240409-013231-marostegui.json
• 01:32 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1190.eqiad.wmnet with reason: Maintenance
• 01:32 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1190.eqiad.wmnet with reason: Maintenance
• 01:32 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2140 (T360332)', diff saved to https://phabricator.wikimedia.org/P59960 and previous config saved to /var/cache/conftool/dbconfig/20240409-013208-arnaudb.json
• 01:32 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1160 (T356166)', diff saved to https://phabricator.wikimedia.org/P59959 and previous config saved to /var/cache/conftool/dbconfig/20240409-013208-marostegui.json
• 01:29 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2140 (T360332)', diff saved to https://phabricator.wikimedia.org/P59958 and previous config saved to /var/cache/conftool/dbconfig/20240409-012949-arnaudb.json
• 01:29 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2140.codfw.wmnet with reason: Maintenance
• 01:29 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2140.codfw.wmnet with reason: Maintenance
• 01:29 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2139.codfw.wmnet with reason: Maintenance
• 01:29 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2139.codfw.wmnet with reason: Maintenance
• 01:29 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2137 (T360332)', diff saved to https://phabricator.wikimedia.org/P59957 and previous config saved to /var/cache/conftool/dbconfig/20240409-012858-arnaudb.json
• 01:17 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1160', diff saved to https://phabricator.wikimedia.org/P59956 and previous config saved to /var/cache/conftool/dbconfig/20240409-011700-marostegui.json
• 01:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2137', diff saved to https://phabricator.wikimedia.org/P59955 and previous config saved to /var/cache/conftool/dbconfig/20240409-011351-arnaudb.json
• 01:01 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1160', diff saved to https://phabricator.wikimedia.org/P59954 and previous config saved to /var/cache/conftool/dbconfig/20240409-010152-marostegui.json
• 00:58 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2137', diff saved to https://phabricator.wikimedia.org/P59953 and previous config saved to /var/cache/conftool/dbconfig/20240409-005843-arnaudb.json
• 00:46 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1160 (T356166)', diff saved to https://phabricator.wikimedia.org/P59952 and previous config saved to /var/cache/conftool/dbconfig/20240409-004645-marostegui.json
• 00:43 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2137 (T360332)', diff saved to https://phabricator.wikimedia.org/P59951 and previous config saved to /var/cache/conftool/dbconfig/20240409-004336-arnaudb.json
• 00:41 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2137 (T360332)', diff saved to https://phabricator.wikimedia.org/P59950 and previous config saved to /var/cache/conftool/dbconfig/20240409-004115-arnaudb.json
• 00:41 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2137.codfw.wmnet with reason: Maintenance
• 00:41 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2137.codfw.wmnet with reason: Maintenance
• 00:40 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2136 (T360332)', diff saved to https://phabricator.wikimedia.org/P59949 and previous config saved to /var/cache/conftool/dbconfig/20240409-004052-arnaudb.json
• 00:25 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2136', diff saved to https://phabricator.wikimedia.org/P59948 and previous config saved to /var/cache/conftool/dbconfig/20240409-002545-arnaudb.json
• 00:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2136', diff saved to https://phabricator.wikimedia.org/P59947 and previous config saved to /var/cache/conftool/dbconfig/20240409-001037-arnaudb.json

2024-04-08

• 23:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2136 (T360332)', diff saved to https://phabricator.wikimedia.org/P59946 and previous config saved to /var/cache/conftool/dbconfig/20240408-235530-arnaudb.json
• 23:53 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2136 (T360332)', diff saved to https://phabricator.wikimedia.org/P59945 and previous config saved to /var/cache/conftool/dbconfig/20240408-235309-arnaudb.json
• 23:53 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2136.codfw.wmnet with reason: Maintenance
• 23:52 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2136.codfw.wmnet with reason: Maintenance
• 23:52 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2119 (T360332)', diff saved to https://phabricator.wikimedia.org/P59944 and previous config saved to /var/cache/conftool/dbconfig/20240408-235247-arnaudb.json
• 23:37 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2119', diff saved to https://phabricator.wikimedia.org/P59943 and previous config saved to /var/cache/conftool/dbconfig/20240408-233739-arnaudb.json
• 23:25 tstarling@deploy1002: Synchronized wmf-config/CommonSettings.php: stop writing to ipblocks table T355034 (duration: 12m 32s)
• 23:22 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2119', diff saved to https://phabricator.wikimedia.org/P59942 and previous config saved to /var/cache/conftool/dbconfig/20240408-232231-arnaudb.json
• 23:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2119 (T360332)', diff saved to https://phabricator.wikimedia.org/P59941 and previous config saved to /var/cache/conftool/dbconfig/20240408-230723-arnaudb.json
• 23:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2119 (T360332)', diff saved to https://phabricator.wikimedia.org/P59940 and previous config saved to /var/cache/conftool/dbconfig/20240408-230502-arnaudb.json
• 23:04 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2119.codfw.wmnet with reason: Maintenance
• 23:04 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2119.codfw.wmnet with reason: Maintenance
• 23:04 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2110.codfw.wmnet with reason: Maintenance
• 23:04 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2110.codfw.wmnet with reason: Maintenance
• 23:04 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2106 (T360332)', diff saved to https://phabricator.wikimedia.org/P59939 and previous config saved to /var/cache/conftool/dbconfig/20240408-230410-arnaudb.json
• 22:49 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2106', diff saved to https://phabricator.wikimedia.org/P59938 and previous config saved to /var/cache/conftool/dbconfig/20240408-224903-arnaudb.json
• 22:33 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2106', diff saved to https://phabricator.wikimedia.org/P59937 and previous config saved to /var/cache/conftool/dbconfig/20240408-223355-arnaudb.json
• 22:18 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2106 (T360332)', diff saved to https://phabricator.wikimedia.org/P59935 and previous config saved to /var/cache/conftool/dbconfig/20240408-221847-arnaudb.json
• 22:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2106 (T360332)', diff saved to https://phabricator.wikimedia.org/P59934 and previous config saved to /var/cache/conftool/dbconfig/20240408-221626-arnaudb.json
• 22:16 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2106.codfw.wmnet with reason: Maintenance
• 22:16 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2106.codfw.wmnet with reason: Maintenance
• 22:15 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2099.codfw.wmnet with reason: Maintenance
• 22:15 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2099.codfw.wmnet with reason: Maintenance
• 22:15 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
• 22:15 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
• 22:15 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1249 (T360332)', diff saved to https://phabricator.wikimedia.org/P59933 and previous config saved to /var/cache/conftool/dbconfig/20240408-221509-arnaudb.json
• 22:00 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1249', diff saved to https://phabricator.wikimedia.org/P59931 and previous config saved to /var/cache/conftool/dbconfig/20240408-220001-arnaudb.json
• 21:55 eileen: config revision changed from fff12a6a to 3c1a0267
• 21:48 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts wdqs1025.eqiad.wmnet
• 21:48 ryankemper@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 21:48 ryankemper@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wdqs1025.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - ryankemper@cumin2002"
• 21:44 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1249', diff saved to https://phabricator.wikimedia.org/P59930 and previous config saved to /var/cache/conftool/dbconfig/20240408-214454-arnaudb.json
• 21:38 eileen: config revision changed from b08eb273 to fff12a6a
• 21:29 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1249 (T360332)', diff saved to https://phabricator.wikimedia.org/P59929 and previous config saved to /var/cache/conftool/dbconfig/20240408-212946-arnaudb.json
• 21:28 ryankemper@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wdqs1025.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - ryankemper@cumin2002"
• 21:27 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 21:27 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 21:26 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1249 (T360332)', diff saved to https://phabricator.wikimedia.org/P59928 and previous config saved to /var/cache/conftool/dbconfig/20240408-212628-arnaudb.json
• 21:26 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1249.eqiad.wmnet with reason: Maintenance
• 21:26 ryankemper@cumin2002: START - Cookbook sre.dns.netbox
• 21:26 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1249.eqiad.wmnet with reason: Maintenance
• 21:26 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1248 (T360332)', diff saved to https://phabricator.wikimedia.org/P59927 and previous config saved to /var/cache/conftool/dbconfig/20240408-212605-arnaudb.json
• 21:20 ryankemper@cumin2002: START - Cookbook sre.hosts.decommission for hosts wdqs1025.eqiad.wmnet
• 21:18 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 21:17 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 21:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1248', diff saved to https://phabricator.wikimedia.org/P59926 and previous config saved to /var/cache/conftool/dbconfig/20240408-211056-arnaudb.json
• 21:10 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 21:10 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 21:01 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 21:01 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 21:01 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 20:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1248', diff saved to https://phabricator.wikimedia.org/P59925 and previous config saved to /var/cache/conftool/dbconfig/20240408-205548-arnaudb.json
• 20:53 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 20:53 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 20:53 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 20:53 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 20:52 ebernhardson@deploy1002: Finished scap: Backport for cirrus: Restore traffic to codfw clusters (duration: 15m 25s)
• 20:42 ebernhardson@deploy1002: ebernhardson: Continuing with sync
• 20:40 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1248 (T360332)', diff saved to https://phabricator.wikimedia.org/P59924 and previous config saved to /var/cache/conftool/dbconfig/20240408-204041-arnaudb.json
• 20:39 ebernhardson@deploy1002: ebernhardson: Backport for cirrus: Restore traffic to codfw clusters synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:37 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1248 (T360332)', diff saved to https://phabricator.wikimedia.org/P59923 and previous config saved to /var/cache/conftool/dbconfig/20240408-203723-arnaudb.json
• 20:37 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1248.eqiad.wmnet with reason: Maintenance
• 20:37 ebernhardson@deploy1002: Started scap: Backport for cirrus: Restore traffic to codfw clusters
• 20:37 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1248.eqiad.wmnet with reason: Maintenance
• 20:37 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1247 (T360332)', diff saved to https://phabricator.wikimedia.org/P59922 and previous config saved to /var/cache/conftool/dbconfig/20240408-203700-arnaudb.json
• 20:29 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 (T360332)', diff saved to https://phabricator.wikimedia.org/P59921 and previous config saved to /var/cache/conftool/dbconfig/20240408-202955-arnaudb.json
• 20:25 urbanecm@deploy1002: Finished scap: Backport for Revert "Mark all autoreviewed edits in PageSaveComplete hook" (T361918 T361940 T361960) (duration: 16m 25s)
• 20:24 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 20:23 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 20:21 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1247', diff saved to https://phabricator.wikimedia.org/P59920 and previous config saved to /var/cache/conftool/dbconfig/20240408-202153-arnaudb.json
• 20:15 urbanecm@deploy1002: urbanecm and matmarex: Continuing with sync
• 20:14 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P59919 and previous config saved to /var/cache/conftool/dbconfig/20240408-201447-arnaudb.json
• 20:11 urbanecm@deploy1002: urbanecm and matmarex: Backport for Revert "Mark all autoreviewed edits in PageSaveComplete hook" (T361918 T361940 T361960) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:09 urbanecm@deploy1002: Started scap: Backport for Revert "Mark all autoreviewed edits in PageSaveComplete hook" (T361918 T361940 T361960)
• 20:06 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1247', diff saved to https://phabricator.wikimedia.org/P59918 and previous config saved to /var/cache/conftool/dbconfig/20240408-200645-arnaudb.json
• 19:59 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P59917 and previous config saved to /var/cache/conftool/dbconfig/20240408-195940-arnaudb.json
• 19:51 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1247 (T360332)', diff saved to https://phabricator.wikimedia.org/P59916 and previous config saved to /var/cache/conftool/dbconfig/20240408-195138-arnaudb.json
• 19:49 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1247 (T360332)', diff saved to https://phabricator.wikimedia.org/P59915 and previous config saved to /var/cache/conftool/dbconfig/20240408-194919-arnaudb.json
• 19:49 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1247.eqiad.wmnet with reason: Maintenance
• 19:48 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1247.eqiad.wmnet with reason: Maintenance
• 19:48 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1245.eqiad.wmnet with reason: Maintenance
• 19:48 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1245.eqiad.wmnet with reason: Maintenance
• 19:48 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1244 (T360332)', diff saved to https://phabricator.wikimedia.org/P59914 and previous config saved to /var/cache/conftool/dbconfig/20240408-194843-arnaudb.json
• 19:44 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 (T360332)', diff saved to https://phabricator.wikimedia.org/P59913 and previous config saved to /var/cache/conftool/dbconfig/20240408-194432-arnaudb.json
• 19:41 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2213 (T360332)', diff saved to https://phabricator.wikimedia.org/P59912 and previous config saved to /var/cache/conftool/dbconfig/20240408-194113-arnaudb.json
• 19:41 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2213.codfw.wmnet with reason: Maintenance
• 19:40 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2213.codfw.wmnet with reason: Maintenance
• 19:40 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2211 (T360332)', diff saved to https://phabricator.wikimedia.org/P59911 and previous config saved to /var/cache/conftool/dbconfig/20240408-194050-arnaudb.json
• 19:33 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P59910 and previous config saved to /var/cache/conftool/dbconfig/20240408-193336-arnaudb.json
• 19:25 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2211', diff saved to https://phabricator.wikimedia.org/P59909 and previous config saved to /var/cache/conftool/dbconfig/20240408-192543-arnaudb.json
• 19:18 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P59908 and previous config saved to /var/cache/conftool/dbconfig/20240408-191828-arnaudb.json
• 19:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2211', diff saved to https://phabricator.wikimedia.org/P59907 and previous config saved to /var/cache/conftool/dbconfig/20240408-191035-arnaudb.json
• 19:03 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1244 (T360332)', diff saved to https://phabricator.wikimedia.org/P59905 and previous config saved to /var/cache/conftool/dbconfig/20240408-190319-arnaudb.json
• 18:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2211 (T360332)', diff saved to https://phabricator.wikimedia.org/P59904 and previous config saved to /var/cache/conftool/dbconfig/20240408-185528-arnaudb.json
• 18:53 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2211 (T360332)', diff saved to https://phabricator.wikimedia.org/P59903 and previous config saved to /var/cache/conftool/dbconfig/20240408-185309-arnaudb.json
• 18:53 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2211.codfw.wmnet with reason: Maintenance
• 18:52 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2211.codfw.wmnet with reason: Maintenance
• 18:52 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2192 (T360332)', diff saved to https://phabricator.wikimedia.org/P59902 and previous config saved to /var/cache/conftool/dbconfig/20240408-185247-arnaudb.json
• 18:37 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2192', diff saved to https://phabricator.wikimedia.org/P59901 and previous config saved to /var/cache/conftool/dbconfig/20240408-183739-arnaudb.json
• 18:22 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2192', diff saved to https://phabricator.wikimedia.org/P59900 and previous config saved to /var/cache/conftool/dbconfig/20240408-182232-arnaudb.json
• 18:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2192 (T360332)', diff saved to https://phabricator.wikimedia.org/P59899 and previous config saved to /var/cache/conftool/dbconfig/20240408-180724-arnaudb.json
• 18:04 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2192 (T360332)', diff saved to https://phabricator.wikimedia.org/P59898 and previous config saved to /var/cache/conftool/dbconfig/20240408-180406-arnaudb.json
• 18:04 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2192.codfw.wmnet with reason: Maintenance
• 18:03 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2192.codfw.wmnet with reason: Maintenance
• 18:03 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2178 (T360332)', diff saved to https://phabricator.wikimedia.org/P59897 and previous config saved to /var/cache/conftool/dbconfig/20240408-180343-arnaudb.json
• 18:03 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1244 (T360332)', diff saved to https://phabricator.wikimedia.org/P59896 and previous config saved to /var/cache/conftool/dbconfig/20240408-180253-arnaudb.json
• 18:02 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1244.eqiad.wmnet with reason: Maintenance
• 18:02 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1244.eqiad.wmnet with reason: Maintenance
• 18:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1243 (T360332)', diff saved to https://phabricator.wikimedia.org/P59895 and previous config saved to /var/cache/conftool/dbconfig/20240408-180231-arnaudb.json
• 17:48 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P59894 and previous config saved to /var/cache/conftool/dbconfig/20240408-174835-arnaudb.json
• 17:47 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1243', diff saved to https://phabricator.wikimedia.org/P59893 and previous config saved to /var/cache/conftool/dbconfig/20240408-174723-arnaudb.json
• 17:33 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P59892 and previous config saved to /var/cache/conftool/dbconfig/20240408-173327-arnaudb.json
• 17:32 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1243', diff saved to https://phabricator.wikimedia.org/P59891 and previous config saved to /var/cache/conftool/dbconfig/20240408-173215-arnaudb.json
• 17:18 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2178 (T360332)', diff saved to https://phabricator.wikimedia.org/P59890 and previous config saved to /var/cache/conftool/dbconfig/20240408-171819-arnaudb.json
• 17:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1243 (T360332)', diff saved to https://phabricator.wikimedia.org/P59889 and previous config saved to /var/cache/conftool/dbconfig/20240408-171707-arnaudb.json
• 17:15 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2178 (T360332)', diff saved to https://phabricator.wikimedia.org/P59888 and previous config saved to /var/cache/conftool/dbconfig/20240408-171502-arnaudb.json
• 17:14 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2178.codfw.wmnet with reason: Maintenance
• 17:14 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1243 (T360332)', diff saved to https://phabricator.wikimedia.org/P59887 and previous config saved to /var/cache/conftool/dbconfig/20240408-171448-arnaudb.json
• 17:14 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2178.codfw.wmnet with reason: Maintenance
• 17:14 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1243.eqiad.wmnet with reason: Maintenance
• 17:14 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2171 (T360332)', diff saved to https://phabricator.wikimedia.org/P59886 and previous config saved to /var/cache/conftool/dbconfig/20240408-171439-arnaudb.json
• 17:14 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1243.eqiad.wmnet with reason: Maintenance
• 17:14 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1242 (T360332)', diff saved to https://phabricator.wikimedia.org/P59885 and previous config saved to /var/cache/conftool/dbconfig/20240408-171425-arnaudb.json
• 16:59 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2171', diff saved to https://phabricator.wikimedia.org/P59884 and previous config saved to /var/cache/conftool/dbconfig/20240408-165931-arnaudb.json
• 16:59 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1242', diff saved to https://phabricator.wikimedia.org/P59883 and previous config saved to /var/cache/conftool/dbconfig/20240408-165917-arnaudb.json
• 16:57 elukey@cumin1002: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching aqs20[08-12]*: Deploy new Truststore - elukey@cumin1002
• 16:45 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1160 (T356166)', diff saved to https://phabricator.wikimedia.org/P59882 and previous config saved to /var/cache/conftool/dbconfig/20240408-164524-marostegui.json
• 16:45 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1160.eqiad.wmnet with reason: Maintenance
• 16:45 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1160.eqiad.wmnet with reason: Maintenance
• 16:44 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2171', diff saved to https://phabricator.wikimedia.org/P59881 and previous config saved to /var/cache/conftool/dbconfig/20240408-164424-arnaudb.json
• 16:44 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1242', diff saved to https://phabricator.wikimedia.org/P59880 and previous config saved to /var/cache/conftool/dbconfig/20240408-164410-arnaudb.json
• 16:32 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 16:32 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 16:32 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 16:32 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 16:29 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2171 (T360332)', diff saved to https://phabricator.wikimedia.org/P59879 and previous config saved to /var/cache/conftool/dbconfig/20240408-162916-arnaudb.json
• 16:29 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1242 (T360332)', diff saved to https://phabricator.wikimedia.org/P59878 and previous config saved to /var/cache/conftool/dbconfig/20240408-162902-arnaudb.json
• 16:27 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2171 (T360332)', diff saved to https://phabricator.wikimedia.org/P59877 and previous config saved to /var/cache/conftool/dbconfig/20240408-162655-arnaudb.json
• 16:26 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2171.codfw.wmnet with reason: Maintenance
• 16:26 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1242 (T360332)', diff saved to https://phabricator.wikimedia.org/P59876 and previous config saved to /var/cache/conftool/dbconfig/20240408-162645-arnaudb.json
• 16:26 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1242.eqiad.wmnet with reason: Maintenance
• 16:26 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2171.codfw.wmnet with reason: Maintenance
• 16:26 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2157 (T360332)', diff saved to https://phabricator.wikimedia.org/P59875 and previous config saved to /var/cache/conftool/dbconfig/20240408-162633-arnaudb.json
• 16:26 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1242.eqiad.wmnet with reason: Maintenance
• 16:26 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1241 (T360332)', diff saved to https://phabricator.wikimedia.org/P59874 and previous config saved to /var/cache/conftool/dbconfig/20240408-162621-arnaudb.json
• 16:19 elukey@cumin1002: START - Cookbook sre.cassandra.roll-restart for nodes matching aqs20[08-12]*: Deploy new Truststore - elukey@cumin1002
• 16:15 elukey: manually dran + restart cassandra-a on aqs2007 - cookbook failed
• 16:15 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts dumpsdata1002.eqiad.wmnet
• 16:14 btullis@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 16:14 btullis@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dumpsdata1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1002"
• 16:11 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P59873 and previous config saved to /var/cache/conftool/dbconfig/20240408-161125-arnaudb.json
• 16:11 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1241', diff saved to https://phabricator.wikimedia.org/P59872 and previous config saved to /var/cache/conftool/dbconfig/20240408-161114-arnaudb.json
• 16:07 btullis@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dumpsdata1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1002"
• 16:06 elukey@cumin1002: END (FAIL) - Cookbook sre.cassandra.roll-restart (exit_code=99) for nodes matching A:aqs-codfw: Deploy new Truststore - elukey@cumin1002
• 16:01 btullis@cumin1002: START - Cookbook sre.dns.netbox
• 15:56 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P59871 and previous config saved to /var/cache/conftool/dbconfig/20240408-155618-arnaudb.json
• 15:56 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1241', diff saved to https://phabricator.wikimedia.org/P59870 and previous config saved to /var/cache/conftool/dbconfig/20240408-155606-arnaudb.json
• 15:48 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 15:47 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 15:43 btullis@cumin1002: START - Cookbook sre.hosts.decommission for hosts dumpsdata1002.eqiad.wmnet
• 15:42 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts dumpsdata1001.eqiad.wmnet
• 15:42 btullis@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 15:42 btullis@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dumpsdata1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1002"
• 15:41 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2157 (T360332)', diff saved to https://phabricator.wikimedia.org/P59869 and previous config saved to /var/cache/conftool/dbconfig/20240408-154110-arnaudb.json
• 15:41 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1241 (T360332)', diff saved to https://phabricator.wikimedia.org/P59868 and previous config saved to /var/cache/conftool/dbconfig/20240408-154059-arnaudb.json
• 15:39 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp3069.esams.wmnet,service=(cdn|ats-be)
• 15:38 btullis@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dumpsdata1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1002"
• 15:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1241 (T360332)', diff saved to https://phabricator.wikimedia.org/P59867 and previous config saved to /var/cache/conftool/dbconfig/20240408-153842-arnaudb.json
• 15:38 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1241.eqiad.wmnet with reason: Maintenance
• 15:38 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1241.eqiad.wmnet with reason: Maintenance
• 15:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1221 (T360332)', diff saved to https://phabricator.wikimedia.org/P59866 and previous config saved to /var/cache/conftool/dbconfig/20240408-153819-arnaudb.json
• 15:37 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2157 (T360332)', diff saved to https://phabricator.wikimedia.org/P59865 and previous config saved to /var/cache/conftool/dbconfig/20240408-153753-arnaudb.json
• 15:37 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2157.codfw.wmnet with reason: Maintenance
• 15:37 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2157.codfw.wmnet with reason: Maintenance
• 15:37 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2128 (T360332)', diff saved to https://phabricator.wikimedia.org/P59864 and previous config saved to /var/cache/conftool/dbconfig/20240408-153730-arnaudb.json
• 15:31 btullis@cumin1002: START - Cookbook sre.dns.netbox
• 15:25 btullis@cumin1002: START - Cookbook sre.hosts.decommission for hosts dumpsdata1001.eqiad.wmnet
• 15:23 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P59863 and previous config saved to /var/cache/conftool/dbconfig/20240408-152311-arnaudb.json
• 15:22 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2128', diff saved to https://phabricator.wikimedia.org/P59862 and previous config saved to /var/cache/conftool/dbconfig/20240408-152221-arnaudb.json
• 15:20 vgutierrez: Uploaded golang-gitlab-wikimedia-sre-qemutest-dev 0.1.0 to apt.wm.o (bookworm)
• 15:17 dancy@deploy1002: Finished deploy [restbase/deploy@c4d19d7]: testing T361608 (duration: 13m 59s)
• 15:15 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3069.esams.wmnet with OS bullseye
• 15:15 elukey@cumin1002: START - Cookbook sre.cassandra.roll-restart for nodes matching A:aqs-codfw: Deploy new Truststore - elukey@cumin1002
• 15:12 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 15:11 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 15:10 elukey: drain and restart cassandra-a on aqs1011 to test the new truststore
• 15:10 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 15:10 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 15:10 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 15:09 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 15:08 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P59860 and previous config saved to /var/cache/conftool/dbconfig/20240408-150803-arnaudb.json
• 15:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2128', diff saved to https://phabricator.wikimedia.org/P59859 and previous config saved to /var/cache/conftool/dbconfig/20240408-150713-arnaudb.json
• 15:04 godog: kill -9 thanos-store on titan1001
• 15:03 dancy@deploy1002: Started deploy [restbase/deploy@c4d19d7]: testing T361608
• 14:52 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1221 (T360332)', diff saved to https://phabricator.wikimedia.org/P59858 and previous config saved to /var/cache/conftool/dbconfig/20240408-145256-arnaudb.json
• 14:52 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2128 (T360332)', diff saved to https://phabricator.wikimedia.org/P59857 and previous config saved to /var/cache/conftool/dbconfig/20240408-145205-arnaudb.json
• 14:51 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3069.esams.wmnet with reason: host reimage
• 14:48 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2128 (T360332)', diff saved to https://phabricator.wikimedia.org/P59856 and previous config saved to /var/cache/conftool/dbconfig/20240408-144847-arnaudb.json
• 14:48 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3069.esams.wmnet with reason: host reimage
• 14:48 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance
• 14:48 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance
• 14:48 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2128.codfw.wmnet with reason: Maintenance
• 14:48 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2128.codfw.wmnet with reason: Maintenance
• 14:48 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2113 (T360332)', diff saved to https://phabricator.wikimedia.org/P59855 and previous config saved to /var/cache/conftool/dbconfig/20240408-144808-arnaudb.json
• 14:47 jayme@deploy1002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 14:47 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1221 (T360332)', diff saved to https://phabricator.wikimedia.org/P59854 and previous config saved to /var/cache/conftool/dbconfig/20240408-144738-arnaudb.json
• 14:47 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
• 14:47 jayme@deploy1002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 14:47 jayme@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 14:47 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
• 14:47 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1221.eqiad.wmnet with reason: Maintenance
• 14:47 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1221.eqiad.wmnet with reason: Maintenance
• 14:47 jayme@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'.
• 14:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1199 (T360332)', diff saved to https://phabricator.wikimedia.org/P59853 and previous config saved to /var/cache/conftool/dbconfig/20240408-144657-arnaudb.json
• 14:44 jayme@deploy1002: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
• 14:43 jayme@deploy1002: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
• 14:43 jayme@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
• 14:42 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 14:42 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 14:42 jayme@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
• 14:41 jayme@deploy1002: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
• 14:40 jayme@deploy1002: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
• 14:39 jayme@deploy1002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
• 14:38 jayme@deploy1002: helmfile [eqiad] START helmfile.d/admin 'apply'.
• 14:37 godog: bounce thanos-query and thanos-store on titan1002 - stuck on high CPU
• 14:33 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2113', diff saved to https://phabricator.wikimedia.org/P59852 and previous config saved to /var/cache/conftool/dbconfig/20240408-143301-arnaudb.json
• 14:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P59851 and previous config saved to /var/cache/conftool/dbconfig/20240408-143149-arnaudb.json
• 14:24 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp3069.esams.wmnet with OS bullseye
• 14:20 jayme@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
• 14:19 sukhe: depool cp3069 to prepare for reimaging: T360430
• 14:19 jayme@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
• 14:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2113', diff saved to https://phabricator.wikimedia.org/P59850 and previous config saved to /var/cache/conftool/dbconfig/20240408-141753-arnaudb.json
• 14:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P59849 and previous config saved to /var/cache/conftool/dbconfig/20240408-141641-arnaudb.json
• 14:06 isaranto@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
• 14:04 isaranto@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
• 14:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2113 (T360332)', diff saved to https://phabricator.wikimedia.org/P59847 and previous config saved to /var/cache/conftool/dbconfig/20240408-140246-arnaudb.json
• 14:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1199 (T360332)', diff saved to https://phabricator.wikimedia.org/P59846 and previous config saved to /var/cache/conftool/dbconfig/20240408-140132-arnaudb.json
• 14:00 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2113 (T360332)', diff saved to https://phabricator.wikimedia.org/P59845 and previous config saved to /var/cache/conftool/dbconfig/20240408-135926-arnaudb.json
• 13:59 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1199 (T360332)', diff saved to https://phabricator.wikimedia.org/P59844 and previous config saved to /var/cache/conftool/dbconfig/20240408-135915-arnaudb.json
• 13:59 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2113.codfw.wmnet with reason: Maintenance
• 13:59 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1199.eqiad.wmnet with reason: Maintenance
• 13:59 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2113.codfw.wmnet with reason: Maintenance
• 13:59 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1199.eqiad.wmnet with reason: Maintenance
• 13:58 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1190 (T360332)', diff saved to https://phabricator.wikimedia.org/P59843 and previous config saved to /var/cache/conftool/dbconfig/20240408-135852-arnaudb.json
• 13:57 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2111.codfw.wmnet with reason: Maintenance
• 13:57 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2111.codfw.wmnet with reason: Maintenance
• 13:56 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2101.codfw.wmnet with reason: Maintenance
• 13:56 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2101.codfw.wmnet with reason: Maintenance
• 13:55 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
• 13:55 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
• 13:55 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1245.eqiad.wmnet with reason: Maintenance
• 13:55 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1245.eqiad.wmnet with reason: Maintenance
• 13:54 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1230 (T360332)', diff saved to https://phabricator.wikimedia.org/P59842 and previous config saved to /var/cache/conftool/dbconfig/20240408-135444-arnaudb.json
• 13:45 Lucas_WMDE: UTC afternoon backport+config window done
• 13:44 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db1207.eqiad.wmnet
• 13:43 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P59841 and previous config saved to /var/cache/conftool/dbconfig/20240408-134345-arnaudb.json
• 13:43 logmsgbot: lucaswerkmeister-wmde@deploy1002 Finished scap: Backport for Enable abusefilter block at bnwiki (T361852) (duration: 28m 46s)
• 13:39 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1230', diff saved to https://phabricator.wikimedia.org/P59840 and previous config saved to /var/cache/conftool/dbconfig/20240408-133936-arnaudb.json
• 13:37 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db1207.eqiad.wmnet
• 13:35 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db1229.eqiad.wmnet
• 13:33 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P{cp[4037,4041,4045,4049].ulsfo.wmnet} and A:cp
• 13:32 logmsgbot: lucaswerkmeister-wmde@deploy1002 lucaswerkmeister-wmde and yahya: Continuing with sync
• 13:28 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P59839 and previous config saved to /var/cache/conftool/dbconfig/20240408-132838-arnaudb.json
• 13:28 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db1229.eqiad.wmnet
• 13:27 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db1175.eqiad.wmnet
• 13:26 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2212.codfw.wmnet with reason: Silence for clone
• 13:26 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2212.codfw.wmnet with reason: Silence for clone
• 13:24 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1230', diff saved to https://phabricator.wikimedia.org/P59838 and previous config saved to /var/cache/conftool/dbconfig/20240408-132429-arnaudb.json
• 13:23 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P{cp[4037,4041,4045,4049].ulsfo.wmnet} and A:cp
• 13:21 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db1175.eqiad.wmnet
• 13:21 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db1241.eqiad.wmnet
• 13:17 logmsgbot: lucaswerkmeister-wmde@deploy1002 lucaswerkmeister-wmde and yahya: Backport for Enable abusefilter block at bnwiki (T361852) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 13:15 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db1241.eqiad.wmnet
• 13:15 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 13:14 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 13:14 logmsgbot: lucaswerkmeister-wmde@deploy1002 Started scap: Backport for Enable abusefilter block at bnwiki (T361852)
• 13:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1190 (T360332)', diff saved to https://phabricator.wikimedia.org/P59837 and previous config saved to /var/cache/conftool/dbconfig/20240408-131331-arnaudb.json
• 13:12 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db1200.eqiad.wmnet
• 13:11 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1190 (T360332)', diff saved to https://phabricator.wikimedia.org/P59836 and previous config saved to /var/cache/conftool/dbconfig/20240408-131113-arnaudb.json
• 13:11 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1190.eqiad.wmnet with reason: Maintenance
• 13:10 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1190.eqiad.wmnet with reason: Maintenance
• 13:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1160 (T360332)', diff saved to https://phabricator.wikimedia.org/P59835 and previous config saved to /var/cache/conftool/dbconfig/20240408-131051-arnaudb.json
• 13:09 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1230 (T360332)', diff saved to https://phabricator.wikimedia.org/P59834 and previous config saved to /var/cache/conftool/dbconfig/20240408-130921-arnaudb.json
• 13:05 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db1200.eqiad.wmnet
• 13:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1230 (T360332)', diff saved to https://phabricator.wikimedia.org/P59833 and previous config saved to /var/cache/conftool/dbconfig/20240408-130543-arnaudb.json
• 13:05 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1230.eqiad.wmnet with reason: Maintenance
• 13:05 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1230.eqiad.wmnet with reason: Maintenance
• 13:04 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1216.eqiad.wmnet with reason: Maintenance
• 13:04 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1216.eqiad.wmnet with reason: Maintenance
• 13:04 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1213 (T360332)', diff saved to https://phabricator.wikimedia.org/P59832 and previous config saved to /var/cache/conftool/dbconfig/20240408-130443-arnaudb.json
• 12:56 elukey: nodetool-a drain + restart of cassandra instances on aqs1010 to pick up the new truststore
• 12:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1160', diff saved to https://phabricator.wikimedia.org/P59831 and previous config saved to /var/cache/conftool/dbconfig/20240408-125543-arnaudb.json
• 12:55 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on aqs1010.eqiad.wmnet with reason: Replace Java Truststore
• 12:55 elukey@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on aqs1010.eqiad.wmnet with reason: Replace Java Truststore
• 12:53 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db1180.eqiad.wmnet
• 12:49 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1213', diff saved to https://phabricator.wikimedia.org/P59830 and previous config saved to /var/cache/conftool/dbconfig/20240408-124935-arnaudb.json
• 12:47 jayme@deploy1002: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
• 12:46 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db1180.eqiad.wmnet
• 12:46 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db1194.eqiad.wmnet
• 12:46 jayme@deploy1002: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
• 12:46 jayme@deploy1002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
• 12:44 jayme@deploy1002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
• 12:27 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db1237.eqiad.wmnet
• 12:25 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1160 (T360332)', diff saved to https://phabricator.wikimedia.org/P59827 and previous config saved to /var/cache/conftool/dbconfig/20240408-122527-arnaudb.json
• 12:22 jayme@deploy1002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
• 12:22 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1160 (T360332)', diff saved to https://phabricator.wikimedia.org/P59826 and previous config saved to /var/cache/conftool/dbconfig/20240408-122209-arnaudb.json
• 12:22 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1160.eqiad.wmnet with reason: Maintenance
• 12:21 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1160.eqiad.wmnet with reason: Maintenance
• 12:19 jayme@deploy1002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
• 12:19 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1213 (T360332)', diff saved to https://phabricator.wikimedia.org/P59825 and previous config saved to /var/cache/conftool/dbconfig/20240408-121920-arnaudb.json
• 12:17 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db1237.eqiad.wmnet
• 12:17 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es1022.eqiad.wmnet
• 12:17 jayme@deploy1002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
• 12:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1213 (T360332)', diff saved to https://phabricator.wikimedia.org/P59824 and previous config saved to /var/cache/conftool/dbconfig/20240408-121642-arnaudb.json
• 12:16 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1213.eqiad.wmnet with reason: Maintenance
• 12:16 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1213.eqiad.wmnet with reason: Maintenance
• 12:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1210 (T360332)', diff saved to https://phabricator.wikimedia.org/P59823 and previous config saved to /var/cache/conftool/dbconfig/20240408-121609-arnaudb.json
• 12:15 jayme@deploy1002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
• 12:14 jayme@deploy1002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
• 12:11 jayme@deploy1002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
• 12:09 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es1022.eqiad.wmnet
• 12:07 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es1031.eqiad.wmnet
• 12:04 jayme@deploy1002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
• 12:02 btullis@cumin1002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host stat1011.eqiad.wmnet
• 12:02 btullis@cumin1002: START - Cookbook sre.hosts.reboot-single for host stat1011.eqiad.wmnet
• 12:01 jayme@deploy1002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
• 12:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1210', diff saved to https://phabricator.wikimedia.org/P59822 and previous config saved to /var/cache/conftool/dbconfig/20240408-120101-arnaudb.json
• 11:57 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es1031.eqiad.wmnet
• 11:57 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es1030.eqiad.wmnet
• 11:49 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es1030.eqiad.wmnet
• 11:49 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es1029.eqiad.wmnet
• 11:45 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1210', diff saved to https://phabricator.wikimedia.org/P59821 and previous config saved to /var/cache/conftool/dbconfig/20240408-114552-arnaudb.json
• 11:42 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es1029.eqiad.wmnet
• 11:35 moritzm: installing glibc security updates on bullseye
• 11:30 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1210 (T360332)', diff saved to https://phabricator.wikimedia.org/P59820 and previous config saved to /var/cache/conftool/dbconfig/20240408-113045-arnaudb.json
• 11:28 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1210 (T360332)', diff saved to https://phabricator.wikimedia.org/P59819 and previous config saved to /var/cache/conftool/dbconfig/20240408-112807-arnaudb.json
• 11:28 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1210.eqiad.wmnet with reason: Maintenance
• 11:27 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1210.eqiad.wmnet with reason: Maintenance
• 11:27 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1200 (T360332)', diff saved to https://phabricator.wikimedia.org/P59818 and previous config saved to /var/cache/conftool/dbconfig/20240408-112744-arnaudb.json
• 11:20 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178 (T360332)', diff saved to https://phabricator.wikimedia.org/P59817 and previous config saved to /var/cache/conftool/dbconfig/20240408-112052-arnaudb.json
• 11:13 hnowlan@deploy1002: helmfile [codfw] [main] DONE helmfile.d/services/mw-jobrunner : sync
• 11:12 hnowlan@deploy1002: helmfile [codfw] [canary] DONE helmfile.d/services/mw-jobrunner : sync
• 11:12 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P59816 and previous config saved to /var/cache/conftool/dbconfig/20240408-111236-arnaudb.json
• 11:12 hnowlan@deploy1002: helmfile [codfw] [canary] START helmfile.d/services/mw-jobrunner : sync
• 11:12 hnowlan@deploy1002: helmfile [codfw] [main] START helmfile.d/services/mw-jobrunner : sync
• 11:11 hnowlan@deploy1002: helmfile [eqiad] [main] DONE helmfile.d/services/mw-jobrunner : sync
• 11:10 hnowlan@deploy1002: helmfile [eqiad] [canary] DONE helmfile.d/services/mw-jobrunner : sync
• 11:09 hnowlan@deploy1002: helmfile [eqiad] [canary] START helmfile.d/services/mw-jobrunner : sync
• 11:09 hnowlan@deploy1002: helmfile [eqiad] [main] START helmfile.d/services/mw-jobrunner : sync
• 11:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P59815 and previous config saved to /var/cache/conftool/dbconfig/20240408-110545-arnaudb.json
• 11:03 btullis: started manual wikidata dump on snapshot1009 for T252396
• 10:57 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P59814 and previous config saved to /var/cache/conftool/dbconfig/20240408-105729-arnaudb.json
• 10:51 Dreamy_Jazz: Starting scan on dewiki for MediaModeration to catch-up on monthly limits - https://wikitech.wikimedia.org/wiki/MediaModeration
• 10:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P59813 and previous config saved to /var/cache/conftool/dbconfig/20240408-105036-arnaudb.json
• 10:49 moritzm: installing postgresql-13 security updates
• 10:42 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1200 (T360332)', diff saved to https://phabricator.wikimedia.org/P59812 and previous config saved to /var/cache/conftool/dbconfig/20240408-104221-arnaudb.json
• 10:39 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1200 (T360332)', diff saved to https://phabricator.wikimedia.org/P59811 and previous config saved to /var/cache/conftool/dbconfig/20240408-103945-arnaudb.json
• 10:39 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1200.eqiad.wmnet with reason: Maintenance
• 10:39 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1200.eqiad.wmnet with reason: Maintenance
• 10:39 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1185 (T360332)', diff saved to https://phabricator.wikimedia.org/P59810 and previous config saved to /var/cache/conftool/dbconfig/20240408-103922-arnaudb.json
• 10:35 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178 (T360332)', diff saved to https://phabricator.wikimedia.org/P59809 and previous config saved to /var/cache/conftool/dbconfig/20240408-103529-arnaudb.json
• 10:33 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1178 (T360332)', diff saved to https://phabricator.wikimedia.org/P59808 and previous config saved to /var/cache/conftool/dbconfig/20240408-103313-arnaudb.json
• 10:33 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1178.eqiad.wmnet with reason: Maintenance
• 10:32 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1178.eqiad.wmnet with reason: Maintenance
• 10:32 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177 (T360332)', diff saved to https://phabricator.wikimedia.org/P59807 and previous config saved to /var/cache/conftool/dbconfig/20240408-103249-arnaudb.json
• 10:24 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P59806 and previous config saved to /var/cache/conftool/dbconfig/20240408-102414-arnaudb.json
• 10:24 Dreamy_Jazz: Starting MediaModeration scanning script (stopped over the weekend due to server instability) - https://wikitech.wikimedia.org/wiki/MediaModeration
• 10:18 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db2128.codfw.wmnet
• 10:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P59805 and previous config saved to /var/cache/conftool/dbconfig/20240408-101741-arnaudb.json
• 10:09 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P59804 and previous config saved to /var/cache/conftool/dbconfig/20240408-100906-arnaudb.json
• 10:07 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db2128.codfw.wmnet
• 10:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P59803 and previous config saved to /var/cache/conftool/dbconfig/20240408-100233-arnaudb.json
• 09:54 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1185 (T360332)', diff saved to https://phabricator.wikimedia.org/P59802 and previous config saved to /var/cache/conftool/dbconfig/20240408-095359-arnaudb.json
• 09:53 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-role (exit_code=0) for role: mariadb::proxy::master
• 09:51 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1185 (T360332)', diff saved to https://phabricator.wikimedia.org/P59801 and previous config saved to /var/cache/conftool/dbconfig/20240408-095123-arnaudb.json
• 09:51 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1185.eqiad.wmnet with reason: Maintenance
• 09:51 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1185.eqiad.wmnet with reason: Maintenance
• 09:51 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T360332)', diff saved to https://phabricator.wikimedia.org/P59800 and previous config saved to /var/cache/conftool/dbconfig/20240408-095100-arnaudb.json
• 09:49 jgiannelos@deploy1002: Finished deploy [restbase/deploy@c4d19d7]: (no justification provided) (duration: 03m 49s)
• 09:47 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177 (T360332)', diff saved to https://phabricator.wikimedia.org/P59799 and previous config saved to /var/cache/conftool/dbconfig/20240408-094726-arnaudb.json
• 09:46 jgiannelos@deploy1002: Started deploy [restbase/deploy@c4d19d7]: (no justification provided)
• 09:45 jayme@deploy1002: helmfile [eqiad] DONE helmfile.d/services/blubberoid: apply
• 09:45 jayme@deploy1002: helmfile [eqiad] START helmfile.d/services/blubberoid: apply
• 09:45 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1177 (T360332)', diff saved to https://phabricator.wikimedia.org/P59798 and previous config saved to /var/cache/conftool/dbconfig/20240408-094510-arnaudb.json
• 09:45 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1177.eqiad.wmnet with reason: Maintenance
• 09:45 jayme@deploy1002: helmfile [codfw] DONE helmfile.d/services/blubberoid: apply
• 09:44 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1177.eqiad.wmnet with reason: Maintenance
• 09:44 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172 (T360332)', diff saved to https://phabricator.wikimedia.org/P59797 and previous config saved to /var/cache/conftool/dbconfig/20240408-094447-arnaudb.json
• 09:44 jayme@deploy1002: helmfile [codfw] START helmfile.d/services/blubberoid: apply
• 09:44 jayme@deploy1002: helmfile [eqiad] DONE helmfile.d/services/apertium: apply
• 09:44 jayme@deploy1002: helmfile [eqiad] START helmfile.d/services/apertium: apply
• 09:43 jayme@deploy1002: helmfile [codfw] DONE helmfile.d/services/apertium: apply
• 09:43 jayme@deploy1002: helmfile [codfw] START helmfile.d/services/apertium: apply
• 09:41 arnaudb@cumin1002: dbctl commit (dc=all): 'db2114 (re)pooling @ 100%: Post clone (src)', diff saved to https://phabricator.wikimedia.org/P59796 and previous config saved to /var/cache/conftool/dbconfig/20240408-094102-arnaudb.json
• 09:37 jmm@cumin2002: START - Cookbook sre.puppet.migrate-role for role: mariadb::proxy::master
• 09:35 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P59795 and previous config saved to /var/cache/conftool/dbconfig/20240408-093552-arnaudb.json
• 09:29 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P59794 and previous config saved to /var/cache/conftool/dbconfig/20240408-092939-arnaudb.json
• 09:25 arnaudb@cumin1002: dbctl commit (dc=all): 'db2114 (re)pooling @ 75%: Post clone (src)', diff saved to https://phabricator.wikimedia.org/P59793 and previous config saved to /var/cache/conftool/dbconfig/20240408-092557-arnaudb.json
• 09:22 jayme@deploy1002: helmfile [staging] DONE helmfile.d/services/apertium: apply
• 09:21 jayme@deploy1002: helmfile [staging] START helmfile.d/services/apertium: apply
• 09:20 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P59792 and previous config saved to /var/cache/conftool/dbconfig/20240408-092045-arnaudb.json
• 09:20 jayme@deploy1002: helmfile [staging] DONE helmfile.d/services/blubberoid: apply
• 09:17 jayme@deploy1002: helmfile [staging] START helmfile.d/services/blubberoid: apply
• 09:17 jayme@deploy1002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
• 09:16 jayme@deploy1002: helmfile [eqiad] START helmfile.d/admin 'apply'.
• 09:16 jayme@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
• 09:16 jayme@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
• 09:14 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 15830
• 09:14 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P59791 and previous config saved to /var/cache/conftool/dbconfig/20240408-091432-arnaudb.json
• 09:10 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 15830
• 09:10 arnaudb@cumin1002: dbctl commit (dc=all): 'db2114 (re)pooling @ 50%: Post clone (src)', diff saved to https://phabricator.wikimedia.org/P59790 and previous config saved to /var/cache/conftool/dbconfig/20240408-091051-arnaudb.json
• 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-role (exit_code=0) for role: mariadb::sanitarium_multiinstance
• 09:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T360332)', diff saved to https://phabricator.wikimedia.org/P59789 and previous config saved to /var/cache/conftool/dbconfig/20240408-090535-arnaudb.json
• 09:03 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1161 (T360332)', diff saved to https://phabricator.wikimedia.org/P59788 and previous config saved to /var/cache/conftool/dbconfig/20240408-090258-arnaudb.json
• 09:02 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
• 09:02 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
• 09:02 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1161.eqiad.wmnet with reason: Maintenance
• 09:02 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1161.eqiad.wmnet with reason: Maintenance
• 08:59 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172 (T360332)', diff saved to https://phabricator.wikimedia.org/P59787 and previous config saved to /var/cache/conftool/dbconfig/20240408-085924-arnaudb.json
• 08:58 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1150.eqiad.wmnet with reason: Maintenance
• 08:58 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1150.eqiad.wmnet with reason: Maintenance
• 08:57 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1172 (T360332)', diff saved to https://phabricator.wikimedia.org/P59786 and previous config saved to /var/cache/conftool/dbconfig/20240408-085708-arnaudb.json
• 08:57 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1172.eqiad.wmnet with reason: Maintenance
• 08:56 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1172.eqiad.wmnet with reason: Maintenance
• 08:55 arnaudb@cumin1002: dbctl commit (dc=all): 'db2114 (re)pooling @ 25%: Post clone (src)', diff saved to https://phabricator.wikimedia.org/P59785 and previous config saved to /var/cache/conftool/dbconfig/20240408-085545-arnaudb.json
• 08:44 jmm@cumin2002: START - Cookbook sre.puppet.migrate-role for role: mariadb::sanitarium_multiinstance
• 08:41 godog: grafana upgrade to 9.5.18 - T361830
• 08:35 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) Will create a clone of db2114.codfw.wmnet onto db2214.codfw.wmnet
• 08:29 dcausse: restarting blazegraph on wdqs1020 (BlazegraphFreeAllocatorsDecreasingRapidly)
• 08:26 brouberol@deploy1002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset: apply
• 08:25 brouberol@deploy1002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset: apply
• 08:24 kartik@deploy1002: Finished scap: Backport for Enable the unified dashboard on the test instance for all languages (T360607) (duration: 15m 47s)
• 08:24 brouberol@deploy1002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset-next: apply
• 08:23 brouberol@deploy1002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset-next: apply
• 08:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Bump db2112 weight T361786', diff saved to https://phabricator.wikimedia.org/P59784 and previous config saved to /var/cache/conftool/dbconfig/20240408-081320-arnaudb.json
• 08:12 kartik@deploy1002: kartik: Continuing with sync
• 08:12 volans: restarted stashbot that had died few minutes ago
• 08:09 arnaudb@cumin1002: dbctl commit (dc=all): 'Promote db2203 to s1 primary T361786', diff saved to https://phabricator.wikimedia.org/P59783 and previous config saved to /var/cache/conftool/dbconfig/20240408-080910-arnaudb.json
• 08:08 arnaudb: Starting s1 codfw failover from db2112 to db2203 - T361786
• 08:08 kartik@deploy1002: Started scap: Backport for Enable the unified dashboard on the test instance for all languages (T360607)
• 07:57 filippo@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/aus-k8s-eqiad-services/jaeger: apply
• 07:57 jayme@deploy1002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
• 07:56 filippo@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/aus-k8s-eqiad-services/jaeger: apply
• 07:56 jayme@deploy1002: helmfile [eqiad] START helmfile.d/admin 'apply'.
• 07:56 jayme@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
• 07:55 jayme@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
• 07:48 kartik@deploy1002: Finished scap: Backport for Add Kartographer Parsoid support to hewikivoyage (T342871 T361025) (duration: 35m 43s)
• 07:47 moritzm: installing util-linux security updates on bullseye/bookworm
• 07:44 marostegui@cumin1002: dbctl commit (dc=all): 'db1156 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P59782 and previous config saved to /var/cache/conftool/dbconfig/20240408-074448-root.json
• 07:40 arnaudb@cumin1002: dbctl commit (dc=all): 'Set db2203 with weight 0 T361786', diff saved to https://phabricator.wikimedia.org/P59781 and previous config saved to /var/cache/conftool/dbconfig/20240408-074006-arnaudb.json
• 07:39 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 37 hosts with reason: Primary switchover s1 T361786
• 07:38 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on 37 hosts with reason: Primary switchover s1 T361786
• 07:35 arnaudb@cumin1002: START - Cookbook sre.mysql.clone Will create a clone of db2114.codfw.wmnet onto db2214.codfw.wmnet
• 07:35 kartik@deploy1002: kartik and ihurbain: Continuing with sync
• 07:32 arnaudb@cumin1002: dbctl commit (dc=all): 'Cloning db2114 in db2214 for T355422', diff saved to https://phabricator.wikimedia.org/P59780 and previous config saved to /var/cache/conftool/dbconfig/20240408-073239-arnaudb.json
• 07:32 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2214.codfw.wmnet with reason: provisionning db2214.codfw.wmnet - T355422
• 07:32 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2214.codfw.wmnet with reason: provisionning db2214.codfw.wmnet - T355422
• 07:32 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2114.codfw.wmnet with reason: provisionning db2214.codfw.wmnet - T355422
• 07:31 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2114.codfw.wmnet with reason: provisionning db2214.codfw.wmnet - T355422
• 07:29 marostegui@cumin1002: dbctl commit (dc=all): 'db1156 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P59779 and previous config saved to /var/cache/conftool/dbconfig/20240408-072942-root.json
• 07:25 kartik@deploy1002: kartik and ihurbain: Backport for Add Kartographer Parsoid support to hewikivoyage (T342871 T361025) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 07:14 marostegui@cumin1002: dbctl commit (dc=all): 'db1156 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P59778 and previous config saved to /var/cache/conftool/dbconfig/20240408-071436-root.json
• 07:12 kartik@deploy1002: Started scap: Backport for Add Kartographer Parsoid support to hewikivoyage (T342871 T361025)
• 06:59 marostegui@cumin1002: dbctl commit (dc=all): 'db1156 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P59777 and previous config saved to /var/cache/conftool/dbconfig/20240408-065931-root.json
• 06:44 marostegui@cumin1002: dbctl commit (dc=all): 'db1156 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P59776 and previous config saved to /var/cache/conftool/dbconfig/20240408-064424-root.json
• 06:29 marostegui@cumin1002: dbctl commit (dc=all): 'db1156 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P59775 and previous config saved to /var/cache/conftool/dbconfig/20240408-062919-root.json
• 06:14 marostegui@cumin1002: dbctl commit (dc=all): 'db1156 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P59774 and previous config saved to /var/cache/conftool/dbconfig/20240408-061413-root.json
• 06:05 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1156', diff saved to https://phabricator.wikimedia.org/P59773 and previous config saved to /var/cache/conftool/dbconfig/20240408-060554-root.json
• 04:02 denisse: Cleaning Prometheus and Thanos-BE log gzips older than 45 days on centrallog2002
• 04:01 denisse: Cleaning Prometheus and Thanos-BE log gzips older than 45 days on centrallog1002

2024-04-06

• 15:33 jhathaway@cumin2002: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM mx-out2001.wikimedia.org
• 15:33 jhathaway@cumin2002: START - Cookbook sre.ganeti.reboot-vm for VM mx-out2001.wikimedia.org
• 03:41 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2198.codfw.wmnet with reason: Maintenance
• 03:41 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2198.codfw.wmnet with reason: Maintenance
• 03:41 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2195 (T360332)', diff saved to https://phabricator.wikimedia.org/P59763 and previous config saved to /var/cache/conftool/dbconfig/20240406-034152-arnaudb.json
• 03:26 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P59762 and previous config saved to /var/cache/conftool/dbconfig/20240406-032644-arnaudb.json
• 03:11 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P59761 and previous config saved to /var/cache/conftool/dbconfig/20240406-031136-arnaudb.json
• 02:56 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2195 (T360332)', diff saved to https://phabricator.wikimedia.org/P59760 and previous config saved to /var/cache/conftool/dbconfig/20240406-025629-arnaudb.json
• 02:54 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2195 (T360332)', diff saved to https://phabricator.wikimedia.org/P59759 and previous config saved to /var/cache/conftool/dbconfig/20240406-025411-arnaudb.json
• 02:54 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2195.codfw.wmnet with reason: Maintenance
• 02:53 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2195.codfw.wmnet with reason: Maintenance
• 02:53 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2181 (T360332)', diff saved to https://phabricator.wikimedia.org/P59758 and previous config saved to /var/cache/conftool/dbconfig/20240406-025348-arnaudb.json
• 02:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P59757 and previous config saved to /var/cache/conftool/dbconfig/20240406-023841-arnaudb.json
• 02:23 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P59756 and previous config saved to /var/cache/conftool/dbconfig/20240406-022333-arnaudb.json
• 02:08 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2181 (T360332)', diff saved to https://phabricator.wikimedia.org/P59755 and previous config saved to /var/cache/conftool/dbconfig/20240406-020826-arnaudb.json
• 02:06 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2181 (T360332)', diff saved to https://phabricator.wikimedia.org/P59754 and previous config saved to /var/cache/conftool/dbconfig/20240406-020608-arnaudb.json
• 02:06 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2181.codfw.wmnet with reason: Maintenance
• 02:05 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2181.codfw.wmnet with reason: Maintenance
• 02:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2167 (T360332)', diff saved to https://phabricator.wikimedia.org/P59753 and previous config saved to /var/cache/conftool/dbconfig/20240406-020545-arnaudb.json
• 01:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P59752 and previous config saved to /var/cache/conftool/dbconfig/20240406-015037-arnaudb.json
• 01:35 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P59751 and previous config saved to /var/cache/conftool/dbconfig/20240406-013530-arnaudb.json
• 01:20 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2167 (T360332)', diff saved to https://phabricator.wikimedia.org/P59750 and previous config saved to /var/cache/conftool/dbconfig/20240406-012021-arnaudb.json
• 01:18 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2167 (T360332)', diff saved to https://phabricator.wikimedia.org/P59749 and previous config saved to /var/cache/conftool/dbconfig/20240406-011803-arnaudb.json
• 01:17 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2167.codfw.wmnet with reason: Maintenance
• 01:17 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2167.codfw.wmnet with reason: Maintenance
• 01:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2166 (T360332)', diff saved to https://phabricator.wikimedia.org/P59748 and previous config saved to /var/cache/conftool/dbconfig/20240406-011740-arnaudb.json
• 01:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P59747 and previous config saved to /var/cache/conftool/dbconfig/20240406-010231-arnaudb.json
• 00:47 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P59746 and previous config saved to /var/cache/conftool/dbconfig/20240406-004724-arnaudb.json
• 00:32 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2166 (T360332)', diff saved to https://phabricator.wikimedia.org/P59745 and previous config saved to /var/cache/conftool/dbconfig/20240406-003216-arnaudb.json
• 00:30 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2166 (T360332)', diff saved to https://phabricator.wikimedia.org/P59744 and previous config saved to /var/cache/conftool/dbconfig/20240406-002958-arnaudb.json
• 00:29 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2166.codfw.wmnet with reason: Maintenance
• 00:29 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2166.codfw.wmnet with reason: Maintenance
• 00:29 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2164 (T360332)', diff saved to https://phabricator.wikimedia.org/P59743 and previous config saved to /var/cache/conftool/dbconfig/20240406-002935-arnaudb.json
• 00:14 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P59742 and previous config saved to /var/cache/conftool/dbconfig/20240406-001428-arnaudb.json

2024-04-05

• 23:59 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P59741 and previous config saved to /var/cache/conftool/dbconfig/20240405-235920-arnaudb.json
• 23:44 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2164 (T360332)', diff saved to https://phabricator.wikimedia.org/P59740 and previous config saved to /var/cache/conftool/dbconfig/20240405-234413-arnaudb.json
• 23:41 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2164 (T360332)', diff saved to https://phabricator.wikimedia.org/P59739 and previous config saved to /var/cache/conftool/dbconfig/20240405-234156-arnaudb.json
• 23:41 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance
• 23:41 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance
• 23:41 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2164.codfw.wmnet with reason: Maintenance
• 23:41 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2164.codfw.wmnet with reason: Maintenance
• 23:41 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2163 (T360332)', diff saved to https://phabricator.wikimedia.org/P59738 and previous config saved to /var/cache/conftool/dbconfig/20240405-234117-arnaudb.json
• 23:26 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P59737 and previous config saved to /var/cache/conftool/dbconfig/20240405-232609-arnaudb.json
• 23:11 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P59736 and previous config saved to /var/cache/conftool/dbconfig/20240405-231102-arnaudb.json
• 22:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2163 (T360332)', diff saved to https://phabricator.wikimedia.org/P59735 and previous config saved to /var/cache/conftool/dbconfig/20240405-225554-arnaudb.json
• 22:53 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2163 (T360332)', diff saved to https://phabricator.wikimedia.org/P59734 and previous config saved to /var/cache/conftool/dbconfig/20240405-225336-arnaudb.json
• 22:53 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2163.codfw.wmnet with reason: Maintenance
• 22:53 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2163.codfw.wmnet with reason: Maintenance
• 22:53 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2162 (T360332)', diff saved to https://phabricator.wikimedia.org/P59733 and previous config saved to /var/cache/conftool/dbconfig/20240405-225313-arnaudb.json
• 22:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2162', diff saved to https://phabricator.wikimedia.org/P59732 and previous config saved to /var/cache/conftool/dbconfig/20240405-223806-arnaudb.json
• 22:22 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2162', diff saved to https://phabricator.wikimedia.org/P59731 and previous config saved to /var/cache/conftool/dbconfig/20240405-222259-arnaudb.json
• 22:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2162 (T360332)', diff saved to https://phabricator.wikimedia.org/P59730 and previous config saved to /var/cache/conftool/dbconfig/20240405-220751-arnaudb.json
• 22:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2162 (T360332)', diff saved to https://phabricator.wikimedia.org/P59729 and previous config saved to /var/cache/conftool/dbconfig/20240405-220533-arnaudb.json
• 22:05 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2162.codfw.wmnet with reason: Maintenance
• 22:05 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2162.codfw.wmnet with reason: Maintenance
• 22:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2161 (T360332)', diff saved to https://phabricator.wikimedia.org/P59728 and previous config saved to /var/cache/conftool/dbconfig/20240405-220510-arnaudb.json
• 21:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2161', diff saved to https://phabricator.wikimedia.org/P59727 and previous config saved to /var/cache/conftool/dbconfig/20240405-215001-arnaudb.json
• 21:34 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2161', diff saved to https://phabricator.wikimedia.org/P59725 and previous config saved to /var/cache/conftool/dbconfig/20240405-213454-arnaudb.json
• 21:19 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2161 (T360332)', diff saved to https://phabricator.wikimedia.org/P59724 and previous config saved to /var/cache/conftool/dbconfig/20240405-211946-arnaudb.json
• 21:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2161 (T360332)', diff saved to https://phabricator.wikimedia.org/P59723 and previous config saved to /var/cache/conftool/dbconfig/20240405-211728-arnaudb.json
• 21:17 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2161.codfw.wmnet with reason: Maintenance
• 21:17 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2161.codfw.wmnet with reason: Maintenance
• 21:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2154 (T360332)', diff saved to https://phabricator.wikimedia.org/P59722 and previous config saved to /var/cache/conftool/dbconfig/20240405-211705-arnaudb.json
• 21:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P59721 and previous config saved to /var/cache/conftool/dbconfig/20240405-210157-arnaudb.json
• 20:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P59720 and previous config saved to /var/cache/conftool/dbconfig/20240405-204650-arnaudb.json
• 20:40 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mx-out2001.wikimedia.org with reason: host reimage
• 20:37 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on mx-out2001.wikimedia.org with reason: host reimage
• 20:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2154 (T360332)', diff saved to https://phabricator.wikimedia.org/P59719 and previous config saved to /var/cache/conftool/dbconfig/20240405-203143-arnaudb.json
• 20:29 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2154 (T360332)', diff saved to https://phabricator.wikimedia.org/P59718 and previous config saved to /var/cache/conftool/dbconfig/20240405-202925-arnaudb.json
• 20:29 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2154.codfw.wmnet with reason: Maintenance
• 20:29 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2154.codfw.wmnet with reason: Maintenance
• 20:29 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2152 (T360332)', diff saved to https://phabricator.wikimedia.org/P59717 and previous config saved to /var/cache/conftool/dbconfig/20240405-202901-arnaudb.json
• 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host mx-out2001.wikimedia.org with OS bookworm
• 20:19 jhathaway@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM mx-out2001.wikimedia.org - jhathaway@cumin2002"
• 20:18 jhathaway@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM mx-out2001.wikimedia.org - jhathaway@cumin2002"
• 20:18 jhathaway@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) mx-out2001.wikimedia.org on all recursors
• 20:18 jhathaway@cumin2002: START - Cookbook sre.dns.wipe-cache mx-out2001.wikimedia.org on all recursors
• 20:18 jhathaway@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 20:18 jhathaway@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM mx-out2001.wikimedia.org - jhathaway@cumin2002"
• 20:16 jhathaway@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM mx-out2001.wikimedia.org - jhathaway@cumin2002"
• 20:15 jhathaway@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mx-out1001.wikimedia.org with OS bookworm
• 20:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2152', diff saved to https://phabricator.wikimedia.org/P59716 and previous config saved to /var/cache/conftool/dbconfig/20240405-201354-arnaudb.json
• 20:13 jhathaway@cumin2002: START - Cookbook sre.dns.netbox
• 20:13 jhathaway@cumin2002: START - Cookbook sre.ganeti.makevm for new host mx-out2001.wikimedia.org
• 20:02 jhathaway@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mx-out1001.wikimedia.org with reason: host reimage
• 19:58 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2152', diff saved to https://phabricator.wikimedia.org/P59715 and previous config saved to /var/cache/conftool/dbconfig/20240405-195847-arnaudb.json
• 19:57 jhathaway@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mx-out1001.wikimedia.org with reason: host reimage
• 19:45 jhathaway@cumin1002: START - Cookbook sre.hosts.reimage for host mx-out1001.wikimedia.org with OS bookworm
• 19:43 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2152 (T360332)', diff saved to https://phabricator.wikimedia.org/P59714 and previous config saved to /var/cache/conftool/dbconfig/20240405-194339-arnaudb.json
• 19:42 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2152 (T360332)', diff saved to https://phabricator.wikimedia.org/P59713 and previous config saved to /var/cache/conftool/dbconfig/20240405-194221-arnaudb.json
• 19:42 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2152.codfw.wmnet with reason: Maintenance
• 19:42 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2152.codfw.wmnet with reason: Maintenance
• 19:41 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2098.codfw.wmnet with reason: Maintenance
• 19:41 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2098.codfw.wmnet with reason: Maintenance
• 19:41 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
• 19:41 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
• 19:40 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1226 (T360332)', diff saved to https://phabricator.wikimedia.org/P59712 and previous config saved to /var/cache/conftool/dbconfig/20240405-194057-arnaudb.json
• 19:40 jhathaway@cumin1002: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=97) for new host mx-out1001.wikimedia.org
• 19:40 jhathaway@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host mx-out1001.wikimedia.org with OS bookworm
• 19:27 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on db1246.eqiad.wmnet with reason: Host down
• 19:27 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on db1246.eqiad.wmnet with reason: Host down
• 19:25 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P59711 and previous config saved to /var/cache/conftool/dbconfig/20240405-192549-arnaudb.json
• 19:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P59710 and previous config saved to /var/cache/conftool/dbconfig/20240405-191042-arnaudb.json
• 19:02 mutante: codesearch - puppet trying to restart hound-search after deploying gerrit:1017179 and gerrit:1016480
• 18:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1226 (T360332)', diff saved to https://phabricator.wikimedia.org/P59709 and previous config saved to /var/cache/conftool/dbconfig/20240405-185533-arnaudb.json
• 18:52 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1226 (T360332)', diff saved to https://phabricator.wikimedia.org/P59708 and previous config saved to /var/cache/conftool/dbconfig/20240405-185216-arnaudb.json
• 18:52 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1226.eqiad.wmnet with reason: Maintenance
• 18:51 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1226.eqiad.wmnet with reason: Maintenance
• 18:51 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1216.eqiad.wmnet with reason: Maintenance
• 18:51 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1216.eqiad.wmnet with reason: Maintenance
• 18:51 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1214 (T360332)', diff saved to https://phabricator.wikimedia.org/P59707 and previous config saved to /var/cache/conftool/dbconfig/20240405-185131-arnaudb.json
• 18:36 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P59706 and previous config saved to /var/cache/conftool/dbconfig/20240405-183623-arnaudb.json
• 18:21 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P59705 and previous config saved to /var/cache/conftool/dbconfig/20240405-182115-arnaudb.json
• 18:13 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1214.eqiad.wmnet
• 18:13 sukhe@cumin2002: START - Cookbook sre.hosts.remove-downtime for db1214.eqiad.wmnet
• 18:06 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1214 (T360332)', diff saved to https://phabricator.wikimedia.org/P59704 and previous config saved to /var/cache/conftool/dbconfig/20240405-180608-arnaudb.json
• 18:03 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1214 (T360332)', diff saved to https://phabricator.wikimedia.org/P59703 and previous config saved to /var/cache/conftool/dbconfig/20240405-180352-arnaudb.json
• 18:03 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1214.eqiad.wmnet with reason: Maintenance
• 18:03 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1214.eqiad.wmnet with reason: Maintenance
• 18:03 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1211 (T360332)', diff saved to https://phabricator.wikimedia.org/P59702 and previous config saved to /var/cache/conftool/dbconfig/20240405-180330-arnaudb.json
• 18:03 dzahn@cumin2002: dbctl commit (dc=all): 'depool db1246', diff saved to https://phabricator.wikimedia.org/P59701 and previous config saved to /var/cache/conftool/dbconfig/20240405-180319-dzahn.json
• 18:01 mutante: depooling db1246 which went down and paged
• 17:47 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P59700 and previous config saved to /var/cache/conftool/dbconfig/20240405-174735-arnaudb.json
• 17:32 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P59699 and previous config saved to /var/cache/conftool/dbconfig/20240405-173227-arnaudb.json
• 17:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1211 (T360332)', diff saved to https://phabricator.wikimedia.org/P59698 and previous config saved to /var/cache/conftool/dbconfig/20240405-171719-arnaudb.json
• 17:15 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1211 (T360332)', diff saved to https://phabricator.wikimedia.org/P59697 and previous config saved to /var/cache/conftool/dbconfig/20240405-171502-arnaudb.json
• 17:14 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1211.eqiad.wmnet with reason: Maintenance
• 17:14 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1211.eqiad.wmnet with reason: Maintenance
• 17:14 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1203 (T360332)', diff saved to https://phabricator.wikimedia.org/P59696 and previous config saved to /var/cache/conftool/dbconfig/20240405-171439-arnaudb.json
• 17:14 jhathaway@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mx-out1001.wikimedia.org with reason: host reimage
• 17:11 jhathaway@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mx-out1001.wikimedia.org with reason: host reimage
• 16:59 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P59695 and previous config saved to /var/cache/conftool/dbconfig/20240405-165931-arnaudb.json
• 16:44 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P59694 and previous config saved to /var/cache/conftool/dbconfig/20240405-164424-arnaudb.json
• 16:29 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1203 (T360332)', diff saved to https://phabricator.wikimedia.org/P59693 and previous config saved to /var/cache/conftool/dbconfig/20240405-162916-arnaudb.json
• 16:27 jhathaway@cumin1002: START - Cookbook sre.hosts.reimage for host mx-out1001.wikimedia.org with OS bookworm
• 16:27 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1203 (T360332)', diff saved to https://phabricator.wikimedia.org/P59692 and previous config saved to /var/cache/conftool/dbconfig/20240405-162700-arnaudb.json
• 16:26 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1203.eqiad.wmnet with reason: Maintenance
• 16:26 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1203.eqiad.wmnet with reason: Maintenance
• 16:26 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193 (T360332)', diff saved to https://phabricator.wikimedia.org/P59691 and previous config saved to /var/cache/conftool/dbconfig/20240405-162637-arnaudb.json
• 16:25 jhathaway@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM mx-out1001.wikimedia.org - jhathaway@cumin1002"
• 16:24 jhathaway@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM mx-out1001.wikimedia.org - jhathaway@cumin1002"
• 16:24 jhathaway@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) mx-out1001.wikimedia.org on all recursors
• 16:24 jhathaway@cumin1002: START - Cookbook sre.dns.wipe-cache mx-out1001.wikimedia.org on all recursors
• 16:24 jhathaway@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 16:24 jhathaway@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM mx-out1001.wikimedia.org - jhathaway@cumin1002"
• 16:18 bking@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 16:18 bking@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 16:11 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193', diff saved to https://phabricator.wikimedia.org/P59689 and previous config saved to /var/cache/conftool/dbconfig/20240405-161130-arnaudb.json
• 16:07 jhathaway@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM mx-out1001.wikimedia.org - jhathaway@cumin1002"
• 16:03 jhathaway@cumin1002: START - Cookbook sre.dns.netbox
• 16:03 jhathaway@cumin1002: START - Cookbook sre.ganeti.makevm for new host mx-out1001.wikimedia.org
• 15:56 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193', diff saved to https://phabricator.wikimedia.org/P59688 and previous config saved to /var/cache/conftool/dbconfig/20240405-155622-arnaudb.json
• 15:45 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudbackup1002-dev.eqiad.wmnet with OS bookworm
• 15:41 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193 (T360332)', diff saved to https://phabricator.wikimedia.org/P59687 and previous config saved to /var/cache/conftool/dbconfig/20240405-154115-arnaudb.json
• 15:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1193 (T360332)', diff saved to https://phabricator.wikimedia.org/P59686 and previous config saved to /var/cache/conftool/dbconfig/20240405-153759-arnaudb.json
• 15:37 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1193.eqiad.wmnet with reason: Maintenance
• 15:37 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1193.eqiad.wmnet with reason: Maintenance
• 15:37 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192 (T360332)', diff saved to https://phabricator.wikimedia.org/P59685 and previous config saved to /var/cache/conftool/dbconfig/20240405-153736-arnaudb.json
• 15:22 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P59684 and previous config saved to /var/cache/conftool/dbconfig/20240405-152228-arnaudb.json
• 15:20 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudbackup1002-dev.eqiad.wmnet with reason: host reimage
• 15:17 andrew@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudbackup1002-dev.eqiad.wmnet with reason: host reimage
• 15:14 aikochou@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
• 15:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P59683 and previous config saved to /var/cache/conftool/dbconfig/20240405-150721-arnaudb.json
• 15:03 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudbackup1002-dev.eqiad.wmnet with OS bookworm
• 14:56 dancy@deploy1002: Installation of scap version "4.75.0" completed for 353 hosts
• 14:55 dancy@deploy1002: Installing scap version "4.75.0" for 353 hosts
• 14:52 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192 (T360332)', diff saved to https://phabricator.wikimedia.org/P59682 and previous config saved to /var/cache/conftool/dbconfig/20240405-145213-arnaudb.json
• 14:49 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1192 (T360332)', diff saved to https://phabricator.wikimedia.org/P59681 and previous config saved to /var/cache/conftool/dbconfig/20240405-144957-arnaudb.json
• 14:49 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1192.eqiad.wmnet with reason: Maintenance
• 14:49 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1192.eqiad.wmnet with reason: Maintenance
• 14:49 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178 (T360332)', diff saved to https://phabricator.wikimedia.org/P59680 and previous config saved to /var/cache/conftool/dbconfig/20240405-144934-arnaudb.json
• 14:36 taavi@cumin1002: conftool action : set/pooled=yes; selector: name=clouddb1016.eqiad.wmnet
• 14:36 taavi@cumin1002: conftool action : set/pooled=no; selector: name=clouddb1016.eqiad.wmnet
• 14:35 taavi@cumin1002: conftool action : set/pooled=yes; selector: name=clouddb1015.eqiad.wmnet
• 14:35 aikochou@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
• 14:34 taavi@cumin1002: conftool action : set/pooled=no; selector: name=clouddb1015.eqiad.wmnet
• 14:34 taavi@cumin1002: conftool action : set/pooled=yes; selector: name=clouddb1014.eqiad.wmnet
• 14:34 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P59679 and previous config saved to /var/cache/conftool/dbconfig/20240405-143427-arnaudb.json
• 14:33 taavi@cumin1002: conftool action : set/pooled=no; selector: name=clouddb1014.eqiad.wmnet
• 14:33 taavi@cumin1002: conftool action : set/pooled=yes; selector: name=clouddb1013.eqiad.wmnet
• 14:29 taavi@cumin1002: conftool action : set/pooled=no; selector: name=clouddb1013.eqiad.wmnet
• 14:26 taavi@cumin1002: conftool action : set/pooled=yes; selector: name=clouddb1020.eqiad.wmnet
• 14:25 taavi@cumin1002: conftool action : set/pooled=no; selector: name=clouddb1020.eqiad.wmnet
• 14:25 taavi@cumin1002: conftool action : set/pooled=yes; selector: name=clouddb1019.eqiad.wmnet
• 14:22 taavi@cumin1002: conftool action : set/pooled=no; selector: name=clouddb1019.eqiad.wmnet
• 14:22 taavi@cumin1002: conftool action : set/pooled=yes; selector: name=clouddb1018.eqiad.wmnet
• 14:20 taavi@cumin1002: conftool action : set/pooled=no; selector: name=clouddb1018.eqiad.wmnet
• 14:19 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P59678 and previous config saved to /var/cache/conftool/dbconfig/20240405-141919-arnaudb.json
• 14:18 taavi@cumin1002: conftool action : set/pooled=yes; selector: name=clouddb1017.eqiad.wmnet
• 14:15 taavi@cumin1002: conftool action : set/pooled=no; selector: name=clouddb1017.eqiad.wmnet
• 14:04 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178 (T360332)', diff saved to https://phabricator.wikimedia.org/P59677 and previous config saved to /var/cache/conftool/dbconfig/20240405-140412-arnaudb.json
• 13:03 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1178 (T360332)', diff saved to https://phabricator.wikimedia.org/P59676 and previous config saved to /var/cache/conftool/dbconfig/20240405-130347-arnaudb.json
• 13:03 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1178.eqiad.wmnet with reason: Maintenance
• 13:03 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1178.eqiad.wmnet with reason: Maintenance
• 13:03 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177 (T360332)', diff saved to https://phabricator.wikimedia.org/P59675 and previous config saved to /var/cache/conftool/dbconfig/20240405-130324-arnaudb.json
• 12:48 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P59674 and previous config saved to /var/cache/conftool/dbconfig/20240405-124816-arnaudb.json
• 12:33 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P59673 and previous config saved to /var/cache/conftool/dbconfig/20240405-123309-arnaudb.json
• 12:32 fabfur@cumin1002: conftool action : set/pooled=yes; selector: name=cp4037.ulsfo.wmnet
• 12:32 fabfur: repool cp4037 (T361845)
• 12:18 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177 (T360332)', diff saved to https://phabricator.wikimedia.org/P59672 and previous config saved to /var/cache/conftool/dbconfig/20240405-121801-arnaudb.json
• 12:10 ayounsi@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts testvm2006.codfw.wmnet
• 12:10 ayounsi@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:10 ayounsi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: testvm2006.codfw.wmnet decommissioned, removing all IPs except the asset tag one - ayounsi@cumin1002"
• 11:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1177 (T360332)', diff saved to https://phabricator.wikimedia.org/P59671 and previous config saved to /var/cache/conftool/dbconfig/20240405-111736-arnaudb.json
• 11:17 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1177.eqiad.wmnet with reason: Maintenance
• 11:17 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1177.eqiad.wmnet with reason: Maintenance
• 11:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172 (T360332)', diff saved to https://phabricator.wikimedia.org/P59670 and previous config saved to /var/cache/conftool/dbconfig/20240405-111713-arnaudb.json
• 11:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P59669 and previous config saved to /var/cache/conftool/dbconfig/20240405-110204-arnaudb.json
• 10:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P59668 and previous config saved to /var/cache/conftool/dbconfig/20240405-104657-arnaudb.json
• 10:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172 (T360332)', diff saved to https://phabricator.wikimedia.org/P59667 and previous config saved to /var/cache/conftool/dbconfig/20240405-103149-arnaudb.json
• 09:51 ayounsi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: testvm2006.codfw.wmnet decommissioned, removing all IPs except the asset tag one - ayounsi@cumin1002"
• 09:38 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
• 09:38 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
• 09:38 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1240.eqiad.wmnet with reason: Maintenance
• 09:38 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1240.eqiad.wmnet with reason: Maintenance
• 09:38 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1239.eqiad.wmnet with reason: Maintenance
• 09:37 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1239.eqiad.wmnet with reason: Maintenance
• 09:37 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1235 (T356166)', diff saved to https://phabricator.wikimedia.org/P59666 and previous config saved to /var/cache/conftool/dbconfig/20240405-093745-marostegui.json
• 09:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1172 (T360332)', diff saved to https://phabricator.wikimedia.org/P59665 and previous config saved to /var/cache/conftool/dbconfig/20240405-093124-arnaudb.json
• 09:31 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1172.eqiad.wmnet with reason: Maintenance
• 09:31 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1172.eqiad.wmnet with reason: Maintenance
• 09:30 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1171.eqiad.wmnet with reason: Maintenance
• 09:30 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1171.eqiad.wmnet with reason: Maintenance
• 09:30 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167 (T360332)', diff saved to https://phabricator.wikimedia.org/P59664 and previous config saved to /var/cache/conftool/dbconfig/20240405-093038-arnaudb.json
• 09:24 ayounsi@cumin1002: START - Cookbook sre.dns.netbox
• 09:22 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1235', diff saved to https://phabricator.wikimedia.org/P59663 and previous config saved to /var/cache/conftool/dbconfig/20240405-092237-marostegui.json
• 09:19 ayounsi@cumin1002: START - Cookbook sre.hosts.decommission for hosts testvm2006.codfw.wmnet
• 09:15 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P59662 and previous config saved to /var/cache/conftool/dbconfig/20240405-091531-arnaudb.json
• 09:07 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1235', diff saved to https://phabricator.wikimedia.org/P59661 and previous config saved to /var/cache/conftool/dbconfig/20240405-090730-marostegui.json
• 09:00 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P59660 and previous config saved to /var/cache/conftool/dbconfig/20240405-090023-arnaudb.json
• 08:52 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1235 (T356166)', diff saved to https://phabricator.wikimedia.org/P59659 and previous config saved to /var/cache/conftool/dbconfig/20240405-085222-marostegui.json
• 08:45 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167 (T360332)', diff saved to https://phabricator.wikimedia.org/P59658 and previous config saved to /var/cache/conftool/dbconfig/20240405-084515-arnaudb.json
• 07:58 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1234', diff saved to https://phabricator.wikimedia.org/P59655 and previous config saved to /var/cache/conftool/dbconfig/20240405-075831-marostegui.json
• 07:56 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1167 (T360332)', diff saved to https://phabricator.wikimedia.org/P59654 and previous config saved to /var/cache/conftool/dbconfig/20240405-075646-arnaudb.json
• 07:56 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
• 07:56 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
• 07:56 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1167.eqiad.wmnet with reason: Maintenance
• 07:56 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1167.eqiad.wmnet with reason: Maintenance
• 07:43 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1234', diff saved to https://phabricator.wikimedia.org/P59653 and previous config saved to /var/cache/conftool/dbconfig/20240405-074323-marostegui.json
• 07:28 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1234 (T356166)', diff saved to https://phabricator.wikimedia.org/P59652 and previous config saved to /var/cache/conftool/dbconfig/20240405-072816-marostegui.json
• 06:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.debug (exit_code=0) for Netbox circuit ID 108
• 06:35 ayounsi@cumin1002: START - Cookbook sre.network.debug for Netbox circuit ID 108
• 04:40 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1234 (T356166)', diff saved to https://phabricator.wikimedia.org/P59651 and previous config saved to /var/cache/conftool/dbconfig/20240405-044048-marostegui.json
• 04:40 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1234.eqiad.wmnet with reason: Maintenance
• 04:40 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1234.eqiad.wmnet with reason: Maintenance
• 04:40 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1232 (T356166)', diff saved to https://phabricator.wikimedia.org/P59650 and previous config saved to /var/cache/conftool/dbconfig/20240405-044025-marostegui.json
• 04:25 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1232', diff saved to https://phabricator.wikimedia.org/P59649 and previous config saved to /var/cache/conftool/dbconfig/20240405-042517-marostegui.json
• 04:10 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1232', diff saved to https://phabricator.wikimedia.org/P59648 and previous config saved to /var/cache/conftool/dbconfig/20240405-041010-marostegui.json
• 03:58 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2220 (T360332)', diff saved to https://phabricator.wikimedia.org/P59647 and previous config saved to /var/cache/conftool/dbconfig/20240405-035829-arnaudb.json
• 03:55 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1232 (T356166)', diff saved to https://phabricator.wikimedia.org/P59646 and previous config saved to /var/cache/conftool/dbconfig/20240405-035503-marostegui.json
• 03:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1232 (T356166)', diff saved to https://phabricator.wikimedia.org/P59645 and previous config saved to /var/cache/conftool/dbconfig/20240405-035353-marostegui.json
• 03:53 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1232.eqiad.wmnet with reason: Maintenance
• 03:53 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1232.eqiad.wmnet with reason: Maintenance
• 03:53 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1228 (T356166)', diff saved to https://phabricator.wikimedia.org/P59644 and previous config saved to /var/cache/conftool/dbconfig/20240405-035331-marostegui.json
• 03:43 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P59643 and previous config saved to /var/cache/conftool/dbconfig/20240405-034322-arnaudb.json
• 03:38 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1228', diff saved to https://phabricator.wikimedia.org/P59642 and previous config saved to /var/cache/conftool/dbconfig/20240405-033823-marostegui.json
• 03:28 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P59641 and previous config saved to /var/cache/conftool/dbconfig/20240405-032814-arnaudb.json
• 03:23 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1228', diff saved to https://phabricator.wikimedia.org/P59640 and previous config saved to /var/cache/conftool/dbconfig/20240405-032316-marostegui.json
• 03:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2220 (T360332)', diff saved to https://phabricator.wikimedia.org/P59639 and previous config saved to /var/cache/conftool/dbconfig/20240405-031307-arnaudb.json
• 03:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2220 (T360332)', diff saved to https://phabricator.wikimedia.org/P59638 and previous config saved to /var/cache/conftool/dbconfig/20240405-031028-arnaudb.json
• 03:10 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2220.codfw.wmnet with reason: Maintenance
• 03:10 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2220.codfw.wmnet with reason: Maintenance
• 03:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2218 (T360332)', diff saved to https://phabricator.wikimedia.org/P59637 and previous config saved to /var/cache/conftool/dbconfig/20240405-031005-arnaudb.json
• 03:08 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1228 (T356166)', diff saved to https://phabricator.wikimedia.org/P59636 and previous config saved to /var/cache/conftool/dbconfig/20240405-030809-marostegui.json
• 02:54 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2218', diff saved to https://phabricator.wikimedia.org/P59635 and previous config saved to /var/cache/conftool/dbconfig/20240405-025458-arnaudb.json
• 02:39 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2218', diff saved to https://phabricator.wikimedia.org/P59634 and previous config saved to /var/cache/conftool/dbconfig/20240405-023949-arnaudb.json
• 02:24 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2218 (T360332)', diff saved to https://phabricator.wikimedia.org/P59633 and previous config saved to /var/cache/conftool/dbconfig/20240405-022442-arnaudb.json
• 02:22 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2218 (T360332)', diff saved to https://phabricator.wikimedia.org/P59632 and previous config saved to /var/cache/conftool/dbconfig/20240405-022201-arnaudb.json
• 02:21 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2218.codfw.wmnet with reason: Maintenance
• 02:21 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2218.codfw.wmnet with reason: Maintenance
• 02:21 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2208 (T360332)', diff saved to https://phabricator.wikimedia.org/P59631 and previous config saved to /var/cache/conftool/dbconfig/20240405-022138-arnaudb.json
• 02:06 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P59630 and previous config saved to /var/cache/conftool/dbconfig/20240405-020630-arnaudb.json
• 01:51 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P59629 and previous config saved to /var/cache/conftool/dbconfig/20240405-015123-arnaudb.json
• 01:36 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2208 (T360332)', diff saved to https://phabricator.wikimedia.org/P59628 and previous config saved to /var/cache/conftool/dbconfig/20240405-013615-arnaudb.json
• 01:33 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2208 (T360332)', diff saved to https://phabricator.wikimedia.org/P59627 and previous config saved to /var/cache/conftool/dbconfig/20240405-013336-arnaudb.json
• 01:33 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2208.codfw.wmnet with reason: Maintenance
• 01:33 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2208.codfw.wmnet with reason: Maintenance
• 01:32 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2198.codfw.wmnet with reason: Maintenance
• 01:32 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2198.codfw.wmnet with reason: Maintenance
• 01:32 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2182 (T360332)', diff saved to https://phabricator.wikimedia.org/P59626 and previous config saved to /var/cache/conftool/dbconfig/20240405-013227-arnaudb.json
• 01:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P59625 and previous config saved to /var/cache/conftool/dbconfig/20240405-011720-arnaudb.json
• 01:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P59623 and previous config saved to /var/cache/conftool/dbconfig/20240405-010212-arnaudb.json
• 00:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1228 (T356166)', diff saved to https://phabricator.wikimedia.org/P59622 and previous config saved to /var/cache/conftool/dbconfig/20240405-005341-marostegui.json
• 00:53 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1228.eqiad.wmnet with reason: Maintenance
• 00:53 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1228.eqiad.wmnet with reason: Maintenance
• 00:53 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1219 (T356166)', diff saved to https://phabricator.wikimedia.org/P59621 and previous config saved to /var/cache/conftool/dbconfig/20240405-005318-marostegui.json
• 00:47 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2182 (T360332)', diff saved to https://phabricator.wikimedia.org/P59620 and previous config saved to /var/cache/conftool/dbconfig/20240405-004705-arnaudb.json
• 00:44 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2182 (T360332)', diff saved to https://phabricator.wikimedia.org/P59619 and previous config saved to /var/cache/conftool/dbconfig/20240405-004428-arnaudb.json
• 00:44 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2182.codfw.wmnet with reason: Maintenance
• 00:44 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2182.codfw.wmnet with reason: Maintenance
• 00:44 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2168 (T360332)', diff saved to https://phabricator.wikimedia.org/P59618 and previous config saved to /var/cache/conftool/dbconfig/20240405-004405-arnaudb.json
• 00:38 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P59617 and previous config saved to /var/cache/conftool/dbconfig/20240405-003810-marostegui.json
• 00:28 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2168', diff saved to https://phabricator.wikimedia.org/P59616 and previous config saved to /var/cache/conftool/dbconfig/20240405-002857-arnaudb.json
• 00:23 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P59615 and previous config saved to /var/cache/conftool/dbconfig/20240405-002303-marostegui.json
• 00:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2168', diff saved to https://phabricator.wikimedia.org/P59614 and previous config saved to /var/cache/conftool/dbconfig/20240405-001350-arnaudb.json
• 00:07 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1219 (T356166)', diff saved to https://phabricator.wikimedia.org/P59613 and previous config saved to /var/cache/conftool/dbconfig/20240405-000755-marostegui.json

2024-04-04

• 23:58 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2168 (T360332)', diff saved to https://phabricator.wikimedia.org/P59612 and previous config saved to /var/cache/conftool/dbconfig/20240404-235843-arnaudb.json
• 23:56 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2168 (T360332)', diff saved to https://phabricator.wikimedia.org/P59611 and previous config saved to /var/cache/conftool/dbconfig/20240404-235606-arnaudb.json
• 23:55 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2168.codfw.wmnet with reason: Maintenance
• 23:55 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2168.codfw.wmnet with reason: Maintenance
• 23:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2159 (T360332)', diff saved to https://phabricator.wikimedia.org/P59610 and previous config saved to /var/cache/conftool/dbconfig/20240404-235543-arnaudb.json
• 23:40 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P59609 and previous config saved to /var/cache/conftool/dbconfig/20240404-234035-arnaudb.json
• 23:25 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P59608 and previous config saved to /var/cache/conftool/dbconfig/20240404-232528-arnaudb.json
• 23:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2159 (T360332)', diff saved to https://phabricator.wikimedia.org/P59606 and previous config saved to /var/cache/conftool/dbconfig/20240404-231020-arnaudb.json
• 23:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2159 (T360332)', diff saved to https://phabricator.wikimedia.org/P59605 and previous config saved to /var/cache/conftool/dbconfig/20240404-230743-arnaudb.json
• 23:07 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance
• 23:07 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance
• 23:07 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2159.codfw.wmnet with reason: Maintenance
• 23:07 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2159.codfw.wmnet with reason: Maintenance
• 23:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2150 (T360332)', diff saved to https://phabricator.wikimedia.org/P59604 and previous config saved to /var/cache/conftool/dbconfig/20240404-230704-arnaudb.json
• 22:51 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2150', diff saved to https://phabricator.wikimedia.org/P59603 and previous config saved to /var/cache/conftool/dbconfig/20240404-225156-arnaudb.json
• 22:41 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
• 22:41 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
• 22:41 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1226 (T355609)', diff saved to https://phabricator.wikimedia.org/P59602 and previous config saved to /var/cache/conftool/dbconfig/20240404-224119-marostegui.json
• 22:36 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2150', diff saved to https://phabricator.wikimedia.org/P59601 and previous config saved to /var/cache/conftool/dbconfig/20240404-223649-arnaudb.json
• 22:26 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P59600 and previous config saved to /var/cache/conftool/dbconfig/20240404-222612-marostegui.json
• 22:21 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2150 (T360332)', diff saved to https://phabricator.wikimedia.org/P59599 and previous config saved to /var/cache/conftool/dbconfig/20240404-222141-arnaudb.json
• 22:19 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2150 (T360332)', diff saved to https://phabricator.wikimedia.org/P59598 and previous config saved to /var/cache/conftool/dbconfig/20240404-221903-arnaudb.json
• 22:18 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2150.codfw.wmnet with reason: Maintenance
• 22:18 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2150.codfw.wmnet with reason: Maintenance
• 22:18 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2122 (T360332)', diff saved to https://phabricator.wikimedia.org/P59597 and previous config saved to /var/cache/conftool/dbconfig/20240404-221839-arnaudb.json
• 22:11 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P59596 and previous config saved to /var/cache/conftool/dbconfig/20240404-221104-marostegui.json
• 22:03 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2122', diff saved to https://phabricator.wikimedia.org/P59595 and previous config saved to /var/cache/conftool/dbconfig/20240404-220331-arnaudb.json
• 21:55 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1226 (T355609)', diff saved to https://phabricator.wikimedia.org/P59594 and previous config saved to /var/cache/conftool/dbconfig/20240404-215557-marostegui.json
• 21:48 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2122', diff saved to https://phabricator.wikimedia.org/P59593 and previous config saved to /var/cache/conftool/dbconfig/20240404-214824-arnaudb.json
• 21:48 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1219 (T356166)', diff saved to https://phabricator.wikimedia.org/P59592 and previous config saved to /var/cache/conftool/dbconfig/20240404-214817-marostegui.json
• 21:48 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1219.eqiad.wmnet with reason: Maintenance
• 21:48 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1219.eqiad.wmnet with reason: Maintenance
• 21:48 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1218 (T356166)', diff saved to https://phabricator.wikimedia.org/P59591 and previous config saved to /var/cache/conftool/dbconfig/20240404-214753-marostegui.json
• 21:33 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2122 (T360332)', diff saved to https://phabricator.wikimedia.org/P59590 and previous config saved to /var/cache/conftool/dbconfig/20240404-213317-arnaudb.json
• 21:32 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1218', diff saved to https://phabricator.wikimedia.org/P59589 and previous config saved to /var/cache/conftool/dbconfig/20240404-213245-marostegui.json
• 21:30 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2122 (T360332)', diff saved to https://phabricator.wikimedia.org/P59588 and previous config saved to /var/cache/conftool/dbconfig/20240404-213031-arnaudb.json
• 21:30 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2122.codfw.wmnet with reason: Maintenance
• 21:30 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2122.codfw.wmnet with reason: Maintenance
• 21:30 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2120 (T360332)', diff saved to https://phabricator.wikimedia.org/P59587 and previous config saved to /var/cache/conftool/dbconfig/20240404-213008-arnaudb.json
• 21:17 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1218', diff saved to https://phabricator.wikimedia.org/P59586 and previous config saved to /var/cache/conftool/dbconfig/20240404-211738-marostegui.json
• 21:15 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2120', diff saved to https://phabricator.wikimedia.org/P59585 and previous config saved to /var/cache/conftool/dbconfig/20240404-211501-arnaudb.json
• 21:12 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1226 (T355609)', diff saved to https://phabricator.wikimedia.org/P59584 and previous config saved to /var/cache/conftool/dbconfig/20240404-211248-marostegui.json
• 21:12 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1226.eqiad.wmnet with reason: Maintenance
• 21:12 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1226.eqiad.wmnet with reason: Maintenance
• 21:02 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1218 (T356166)', diff saved to https://phabricator.wikimedia.org/P59583 and previous config saved to /var/cache/conftool/dbconfig/20240404-210230-marostegui.json
• 20:59 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2120', diff saved to https://phabricator.wikimedia.org/P59582 and previous config saved to /var/cache/conftool/dbconfig/20240404-205953-arnaudb.json
• 20:44 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2120 (T360332)', diff saved to https://phabricator.wikimedia.org/P59581 and previous config saved to /var/cache/conftool/dbconfig/20240404-204446-arnaudb.json
• 20:42 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2120 (T360332)', diff saved to https://phabricator.wikimedia.org/P59580 and previous config saved to /var/cache/conftool/dbconfig/20240404-204204-arnaudb.json
• 20:41 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2120.codfw.wmnet with reason: Maintenance
• 20:41 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2120.codfw.wmnet with reason: Maintenance
• 20:41 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2108 (T360332)', diff saved to https://phabricator.wikimedia.org/P59579 and previous config saved to /var/cache/conftool/dbconfig/20240404-204141-arnaudb.json
• 20:33 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1216.eqiad.wmnet with reason: Maintenance
• 20:33 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1216.eqiad.wmnet with reason: Maintenance
• 20:33 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1214 (T355609)', diff saved to https://phabricator.wikimedia.org/P59578 and previous config saved to /var/cache/conftool/dbconfig/20240404-203302-marostegui.json
• 20:26 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2108', diff saved to https://phabricator.wikimedia.org/P59577 and previous config saved to /var/cache/conftool/dbconfig/20240404-202634-arnaudb.json
• 20:17 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P59576 and previous config saved to /var/cache/conftool/dbconfig/20240404-201755-marostegui.json
• 20:11 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2108', diff saved to https://phabricator.wikimedia.org/P59575 and previous config saved to /var/cache/conftool/dbconfig/20240404-201126-arnaudb.json
• 20:02 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P59574 and previous config saved to /var/cache/conftool/dbconfig/20240404-200247-marostegui.json
• 19:56 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2108 (T360332)', diff saved to https://phabricator.wikimedia.org/P59573 and previous config saved to /var/cache/conftool/dbconfig/20240404-195615-arnaudb.json
• 19:53 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2108 (T360332)', diff saved to https://phabricator.wikimedia.org/P59572 and previous config saved to /var/cache/conftool/dbconfig/20240404-195333-arnaudb.json
• 19:53 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2108.codfw.wmnet with reason: Maintenance
• 19:53 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2108.codfw.wmnet with reason: Maintenance
• 19:52 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2098.codfw.wmnet with reason: Maintenance
• 19:52 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2098.codfw.wmnet with reason: Maintenance
• 19:51 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
• 19:51 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
• 19:51 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1236 (T360332)', diff saved to https://phabricator.wikimedia.org/P59571 and previous config saved to /var/cache/conftool/dbconfig/20240404-195138-arnaudb.json
• 19:47 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1214 (T355609)', diff saved to https://phabricator.wikimedia.org/P59570 and previous config saved to /var/cache/conftool/dbconfig/20240404-194739-marostegui.json
• 19:36 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1236', diff saved to https://phabricator.wikimedia.org/P59569 and previous config saved to /var/cache/conftool/dbconfig/20240404-193631-arnaudb.json
• 19:21 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1236', diff saved to https://phabricator.wikimedia.org/P59568 and previous config saved to /var/cache/conftool/dbconfig/20240404-192123-arnaudb.json
• 19:11 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp4052.ulsfo.wmnet,service=(cdn|ats-be)
• 19:10 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4052.ulsfo.wmnet with OS bullseye
• 19:06 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1236 (T360332)', diff saved to https://phabricator.wikimedia.org/P59567 and previous config saved to /var/cache/conftool/dbconfig/20240404-190616-arnaudb.json
• 19:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1236 (T360332)', diff saved to https://phabricator.wikimedia.org/P59566 and previous config saved to /var/cache/conftool/dbconfig/20240404-190146-arnaudb.json
• 19:01 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1236.eqiad.wmnet with reason: Maintenance
• 19:01 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1236.eqiad.wmnet with reason: Maintenance
• 19:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1227 (T360332)', diff saved to https://phabricator.wikimedia.org/P59565 and previous config saved to /var/cache/conftool/dbconfig/20240404-190123-arnaudb.json
• 18:55 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1214 (T355609)', diff saved to https://phabricator.wikimedia.org/P59564 and previous config saved to /var/cache/conftool/dbconfig/20240404-185458-marostegui.json
• 18:54 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1214.eqiad.wmnet with reason: Maintenance
• 18:54 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1214.eqiad.wmnet with reason: Maintenance
• 18:54 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1211 (T355609)', diff saved to https://phabricator.wikimedia.org/P59563 and previous config saved to /var/cache/conftool/dbconfig/20240404-185436-marostegui.json
• 18:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1218 (T356166)', diff saved to https://phabricator.wikimedia.org/P59562 and previous config saved to /var/cache/conftool/dbconfig/20240404-185319-marostegui.json
• 18:53 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1218.eqiad.wmnet with reason: Maintenance
• 18:53 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1218.eqiad.wmnet with reason: Maintenance
• 18:52 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1207 (T356166)', diff saved to https://phabricator.wikimedia.org/P59561 and previous config saved to /var/cache/conftool/dbconfig/20240404-185256-marostegui.json
• 18:49 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage
• 18:46 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage
• 18:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P59560 and previous config saved to /var/cache/conftool/dbconfig/20240404-184616-arnaudb.json
• 18:41 denisse@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on webperf2003.codfw.wmnet,webperf1003.eqiad.wmnet with reason: Downtiming the webperf hosts part of the cergen to CFSSL migration - T360414
• 18:40 denisse@cumin2002: START - Cookbook sre.hosts.downtime for 0:30:00 on webperf2003.codfw.wmnet,webperf1003.eqiad.wmnet with reason: Downtiming the webperf hosts part of the cergen to CFSSL migration - T360414
• 18:39 denisse@cumin2002: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 0:30:00 on performance.wikimedia.org with reason: Downtiming the webperf hosts part of the cergen to CFSSL migration - T360414
• 18:39 denisse@cumin2002: START - Cookbook sre.hosts.downtime for 0:30:00 on performance.wikimedia.org with reason: Downtiming the webperf hosts part of the cergen to CFSSL migration - T360414
• 18:39 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P59559 and previous config saved to /var/cache/conftool/dbconfig/20240404-183928-marostegui.json
• 18:37 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P59558 and previous config saved to /var/cache/conftool/dbconfig/20240404-183748-marostegui.json
• 18:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P59557 and previous config saved to /var/cache/conftool/dbconfig/20240404-183108-arnaudb.json
• 18:31 denisse: Disabling Puppet on the webperf hosts part of the cergen to CFSSL migration - T360414
• 18:24 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P59556 and previous config saved to /var/cache/conftool/dbconfig/20240404-182421-marostegui.json
• 18:24 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp4052.ulsfo.wmnet with OS bullseye
• 18:23 sukhe@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4052.ulsfo.wmnet with OS bullseye
• 18:22 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P59555 and previous config saved to /var/cache/conftool/dbconfig/20240404-182241-marostegui.json
• 18:19 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudbackup1002-dev.eqiad.wmnet with OS bookworm
• 18:16 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp4052.ulsfo.wmnet with OS bullseye
• 18:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1227 (T360332)', diff saved to https://phabricator.wikimedia.org/P59554 and previous config saved to /var/cache/conftool/dbconfig/20240404-181601-arnaudb.json
• 18:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1227 (T360332)', diff saved to https://phabricator.wikimedia.org/P59553 and previous config saved to /var/cache/conftool/dbconfig/20240404-181326-arnaudb.json
• 18:13 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1227.eqiad.wmnet with reason: Maintenance
• 18:13 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1227.eqiad.wmnet with reason: Maintenance
• 18:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1202 (T360332)', diff saved to https://phabricator.wikimedia.org/P59552 and previous config saved to /var/cache/conftool/dbconfig/20240404-181303-arnaudb.json
• 18:09 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1211 (T355609)', diff saved to https://phabricator.wikimedia.org/P59551 and previous config saved to /var/cache/conftool/dbconfig/20240404-180913-marostegui.json
• 18:07 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1207 (T356166)', diff saved to https://phabricator.wikimedia.org/P59550 and previous config saved to /var/cache/conftool/dbconfig/20240404-180733-marostegui.json
• 18:04 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudbackup1002-dev.eqiad.wmnet with reason: host reimage
• 18:01 andrew@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudbackup1002-dev.eqiad.wmnet with reason: host reimage
• 17:57 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P59549 and previous config saved to /var/cache/conftool/dbconfig/20240404-175756-arnaudb.json
• 17:49 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudbackup1002-dev.eqiad.wmnet with OS bookworm
• 17:48 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudbackup1001-dev.eqiad.wmnet with OS bookworm
• 17:48 sukhe: depool cp4052 to prepare for reimaging
• 17:46 sukhe@cumin1002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp4052.ulsfo.wmnet
• 17:42 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P59548 and previous config saved to /var/cache/conftool/dbconfig/20240404-174246-arnaudb.json
• 17:37 sukhe@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp4052.ulsfo.wmnet
• 17:36 ebernhardson@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 17:35 ebernhardson@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 17:29 moritzm: installing isl bugfix updates from Bookworm point release
• 17:27 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1202 (T360332)', diff saved to https://phabricator.wikimedia.org/P59547 and previous config saved to /var/cache/conftool/dbconfig/20240404-172739-arnaudb.json
• 17:24 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1202 (T360332)', diff saved to https://phabricator.wikimedia.org/P59546 and previous config saved to /var/cache/conftool/dbconfig/20240404-172408-arnaudb.json
• 17:24 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1202.eqiad.wmnet with reason: Maintenance
• 17:23 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1202.eqiad.wmnet with reason: Maintenance
• 17:23 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1194 (T360332)', diff saved to https://phabricator.wikimedia.org/P59545 and previous config saved to /var/cache/conftool/dbconfig/20240404-172343-arnaudb.json
• 17:22 moritzm: installing qemu security updates on bookworm
• 17:19 jforrester@deploy1002: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
• 17:17 jforrester@deploy1002: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
• 17:17 jforrester@deploy1002: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
• 17:15 jforrester@deploy1002: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
• 17:13 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1211 (T355609)', diff saved to https://phabricator.wikimedia.org/P59544 and previous config saved to /var/cache/conftool/dbconfig/20240404-171347-marostegui.json
• 17:13 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1211.eqiad.wmnet with reason: Maintenance
• 17:13 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1211.eqiad.wmnet with reason: Maintenance
• 17:13 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1203 (T355609)', diff saved to https://phabricator.wikimedia.org/P59543 and previous config saved to /var/cache/conftool/dbconfig/20240404-171324-marostegui.json
• 17:13 jforrester@deploy1002: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
• 17:12 jforrester@deploy1002: helmfile [staging] START helmfile.d/services/wikifunctions: apply
• 17:09 fabfur@cumin1002: conftool action : set/pooled=yes; selector: name=cp3068.esams.wmnet
• 17:08 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P59542 and previous config saved to /var/cache/conftool/dbconfig/20240404-170836-arnaudb.json
• 16:58 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P59541 and previous config saved to /var/cache/conftool/dbconfig/20240404-165816-marostegui.json
• 16:58 pmiazga: T355281 executed “mwscript extensions/WikimediaMaintenance/addWiki.php --wiki=aawiki --skipclusters=main,echo,growth,mediamoderation,extstore en wikipedia test2wiki test2.wikipedia.beta.wmcloud.org” on deployment-deploy03.deployment-prep
• 16:55 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3068.esams.wmnet with OS bullseye
• 16:54 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudbackup1001-dev.eqiad.wmnet with reason: host reimage
• 16:53 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P59540 and previous config saved to /var/cache/conftool/dbconfig/20240404-165328-arnaudb.json
• 16:51 andrew@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudbackup1001-dev.eqiad.wmnet with reason: host reimage
• 16:43 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P59539 and previous config saved to /var/cache/conftool/dbconfig/20240404-164309-marostegui.json
• 16:42 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudbackup1001-dev.eqiad.wmnet with OS bookworm
• 16:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1194 (T360332)', diff saved to https://phabricator.wikimedia.org/P59538 and previous config saved to /var/cache/conftool/dbconfig/20240404-163819-arnaudb.json
• 16:35 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1194 (T360332)', diff saved to https://phabricator.wikimedia.org/P59537 and previous config saved to /var/cache/conftool/dbconfig/20240404-163549-arnaudb.json
• 16:35 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1194.eqiad.wmnet with reason: Maintenance
• 16:35 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1194.eqiad.wmnet with reason: Maintenance
• 16:35 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1191 (T360332)', diff saved to https://phabricator.wikimedia.org/P59536 and previous config saved to /var/cache/conftool/dbconfig/20240404-163526-arnaudb.json
• 16:32 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3068.esams.wmnet with reason: host reimage
• 16:31 arnaudb@cumin1002: dbctl commit (dc=all): 'db2213 (re)pooling @ 100%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59535 and previous config saved to /var/cache/conftool/dbconfig/20240404-163107-arnaudb.json
• 16:30 arnaudb@cumin1002: dbctl commit (dc=all): 'db2207 (re)pooling @ 100%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59534 and previous config saved to /var/cache/conftool/dbconfig/20240404-163053-arnaudb.json
• 16:28 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1203 (T355609)', diff saved to https://phabricator.wikimedia.org/P59533 and previous config saved to /var/cache/conftool/dbconfig/20240404-162801-marostegui.json
• 16:27 fabfur@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3068.esams.wmnet with reason: host reimage
• 16:20 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P59532 and previous config saved to /var/cache/conftool/dbconfig/20240404-162019-arnaudb.json
• 16:16 arnaudb@cumin1002: dbctl commit (dc=all): 'db2213 (re)pooling @ 75%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59531 and previous config saved to /var/cache/conftool/dbconfig/20240404-161601-arnaudb.json
• 16:15 arnaudb@cumin1002: dbctl commit (dc=all): 'db2207 (re)pooling @ 75%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59530 and previous config saved to /var/cache/conftool/dbconfig/20240404-161547-arnaudb.json
• 16:10 jforrester@deploy1002: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
• 16:09 jforrester@deploy1002: helmfile [staging] START helmfile.d/services/wikifunctions: apply
• 16:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P59529 and previous config saved to /var/cache/conftool/dbconfig/20240404-160508-arnaudb.json
• 16:04 jforrester@deploy1002: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
• 16:04 fabfur@cumin1002: START - Cookbook sre.hosts.reimage for host cp3068.esams.wmnet with OS bullseye
• 16:03 jforrester@deploy1002: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
• 16:03 jforrester@deploy1002: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
• 16:02 jforrester@deploy1002: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
• 16:01 jforrester@deploy1002: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
• 16:01 jforrester@deploy1002: helmfile [staging] START helmfile.d/services/wikifunctions: apply
• 16:00 arnaudb@cumin1002: dbctl commit (dc=all): 'db2213 (re)pooling @ 50%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59528 and previous config saved to /var/cache/conftool/dbconfig/20240404-160055-arnaudb.json
• 16:00 arnaudb@cumin1002: dbctl commit (dc=all): 'db2207 (re)pooling @ 50%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59527 and previous config saved to /var/cache/conftool/dbconfig/20240404-160041-arnaudb.json
• 15:55 fabfur@cumin1002: conftool action : set/pooled=no; selector: name=cp3068.esams.wmnet
• 15:54 fabfur: depooling cp3068 for reimage (T360430)
• 15:54 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1207 (T356166)', diff saved to https://phabricator.wikimedia.org/P59526 and previous config saved to /var/cache/conftool/dbconfig/20240404-155420-marostegui.json
• 15:54 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1207.eqiad.wmnet with reason: Maintenance
• 15:54 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1207.eqiad.wmnet with reason: Maintenance
• 15:53 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1206 (T356166)', diff saved to https://phabricator.wikimedia.org/P59525 and previous config saved to /var/cache/conftool/dbconfig/20240404-155357-marostegui.json
• 15:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1191 (T360332)', diff saved to https://phabricator.wikimedia.org/P59524 and previous config saved to /var/cache/conftool/dbconfig/20240404-155000-arnaudb.json
• 15:47 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1191 (T360332)', diff saved to https://phabricator.wikimedia.org/P59523 and previous config saved to /var/cache/conftool/dbconfig/20240404-154730-arnaudb.json
• 15:47 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1191.eqiad.wmnet with reason: Maintenance
• 15:47 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1191.eqiad.wmnet with reason: Maintenance
• 15:47 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T360332)', diff saved to https://phabricator.wikimedia.org/P59522 and previous config saved to /var/cache/conftool/dbconfig/20240404-154707-arnaudb.json
• 15:45 arnaudb@cumin1002: dbctl commit (dc=all): 'db2213 (re)pooling @ 25%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59521 and previous config saved to /var/cache/conftool/dbconfig/20240404-154549-arnaudb.json
• 15:45 arnaudb@cumin1002: dbctl commit (dc=all): 'db2207 (re)pooling @ 25%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59520 and previous config saved to /var/cache/conftool/dbconfig/20240404-154535-arnaudb.json
• 15:45 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.debug (exit_code=0) for Netbox circuit ID 108
• 15:45 ayounsi@cumin1002: START - Cookbook sre.network.debug for Netbox circuit ID 108
• 15:38 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1206', diff saved to https://phabricator.wikimedia.org/P59519 and previous config saved to /var/cache/conftool/dbconfig/20240404-153850-marostegui.json
• 15:36 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1203 (T355609)', diff saved to https://phabricator.wikimedia.org/P59518 and previous config saved to /var/cache/conftool/dbconfig/20240404-153626-marostegui.json
• 15:36 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on db2214.codfw.wmnet with reason: depooled, see T361851
• 15:36 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1203.eqiad.wmnet with reason: Maintenance
• 15:36 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on db2214.codfw.wmnet with reason: depooled, see T361851
• 15:36 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1203.eqiad.wmnet with reason: Maintenance
• 15:36 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193 (T355609)', diff saved to https://phabricator.wikimedia.org/P59517 and previous config saved to /var/cache/conftool/dbconfig/20240404-153603-marostegui.json
• 15:32 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P59516 and previous config saved to /var/cache/conftool/dbconfig/20240404-153200-arnaudb.json
• 15:30 arnaudb@cumin1002: dbctl commit (dc=all): 'db2213 (re)pooling @ 16%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59515 and previous config saved to /var/cache/conftool/dbconfig/20240404-153043-arnaudb.json
• 15:30 arnaudb@cumin1002: dbctl commit (dc=all): 'db2207 (re)pooling @ 16%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59514 and previous config saved to /var/cache/conftool/dbconfig/20240404-153030-arnaudb.json
• 15:30 sukhe@cumin2002: dbctl commit (dc=all): 'depool db2214', diff saved to https://phabricator.wikimedia.org/P59513 and previous config saved to /var/cache/conftool/dbconfig/20240404-153023-sukhe.json
• 15:23 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1206', diff saved to https://phabricator.wikimedia.org/P59512 and previous config saved to /var/cache/conftool/dbconfig/20240404-152341-marostegui.json
• 15:23 jgiannelos@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
• 15:22 jgiannelos@deploy1002: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
• 15:22 logmsgbot: dreamyjazz Deployed security patch for T361296
• 15:20 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193', diff saved to https://phabricator.wikimedia.org/P59511 and previous config saved to /var/cache/conftool/dbconfig/20240404-152056-marostegui.json
• 15:18 jgiannelos@deploy1002: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
• 15:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P59510 and previous config saved to /var/cache/conftool/dbconfig/20240404-151653-arnaudb.json
• 15:16 jgiannelos@deploy1002: helmfile [codfw] START helmfile.d/services/mobileapps: apply
• 15:16 jgiannelos@deploy1002: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
• 15:16 jgiannelos@deploy1002: helmfile [staging] START helmfile.d/services/mobileapps: apply
• 15:15 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 (re)pooling @ 100%: Post clone repool (src)', diff saved to https://phabricator.wikimedia.org/P59509 and previous config saved to /var/cache/conftool/dbconfig/20240404-151547-arnaudb.json
• 15:15 jgiannelos@deploy1002: helmfile [codfw] START helmfile.d/services/mobileapps: apply
• 15:15 arnaudb@cumin1002: dbctl commit (dc=all): 'db2213 (re)pooling @ 8%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59508 and previous config saved to /var/cache/conftool/dbconfig/20240404-151537-arnaudb.json
• 15:15 arnaudb@cumin1002: dbctl commit (dc=all): 'db2207 (re)pooling @ 8%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59507 and previous config saved to /var/cache/conftool/dbconfig/20240404-151524-arnaudb.json
• 15:11 arnaudb@cumin1002: dbctl commit (dc=all): 'db2107 (re)pooling @ 100%: Post clone repool (src)', diff saved to https://phabricator.wikimedia.org/P59506 and previous config saved to /var/cache/conftool/dbconfig/20240404-151123-arnaudb.json
• 15:10 herron: beginning rolling hardware upgrades on titan200[12] T361229
• 15:08 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1206 (T356166)', diff saved to https://phabricator.wikimedia.org/P59505 and previous config saved to /var/cache/conftool/dbconfig/20240404-150833-marostegui.json
• 15:08 jgiannelos@deploy1002: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
• 15:08 jgiannelos@deploy1002: helmfile [staging] START helmfile.d/services/mobileapps: apply
• 15:05 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193', diff saved to https://phabricator.wikimedia.org/P59504 and previous config saved to /var/cache/conftool/dbconfig/20240404-150549-marostegui.json
• 15:03 logmsgbot: dreamyjazz Deployed security patch for T361295
• 15:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T360332)', diff saved to https://phabricator.wikimedia.org/P59503 and previous config saved to /var/cache/conftool/dbconfig/20240404-150145-arnaudb.json
• 15:00 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 (re)pooling @ 70%: Post clone repool (src)', diff saved to https://phabricator.wikimedia.org/P59502 and previous config saved to /var/cache/conftool/dbconfig/20240404-150041-arnaudb.json
• 15:00 arnaudb@cumin1002: dbctl commit (dc=all): 'db2213 (re)pooling @ 4%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59501 and previous config saved to /var/cache/conftool/dbconfig/20240404-150032-arnaudb.json
• 15:00 arnaudb@cumin1002: dbctl commit (dc=all): 'db2207 (re)pooling @ 4%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59500 and previous config saved to /var/cache/conftool/dbconfig/20240404-150017-arnaudb.json
• 14:57 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1174 (T360332)', diff saved to https://phabricator.wikimedia.org/P59499 and previous config saved to /var/cache/conftool/dbconfig/20240404-145714-arnaudb.json
• 14:57 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1174.eqiad.wmnet with reason: Maintenance
• 14:56 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1174.eqiad.wmnet with reason: Maintenance
• 14:56 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1171.eqiad.wmnet with reason: Maintenance
• 14:56 arnaudb@cumin1002: dbctl commit (dc=all): 'db2107 (re)pooling @ 70%: Post clone repool (src)', diff saved to https://phabricator.wikimedia.org/P59498 and previous config saved to /var/cache/conftool/dbconfig/20240404-145617-arnaudb.json
• 14:56 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1171.eqiad.wmnet with reason: Maintenance
• 14:56 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1170 (T360332)', diff saved to https://phabricator.wikimedia.org/P59497 and previous config saved to /var/cache/conftool/dbconfig/20240404-145613-arnaudb.json
• 14:50 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193 (T355609)', diff saved to https://phabricator.wikimedia.org/P59496 and previous config saved to /var/cache/conftool/dbconfig/20240404-145041-marostegui.json
• 14:48 dreamyjazz@deploy1002: Finished scap: Backport for Remove sk translation of centralauth-rightslog-name (T361695) (duration: 14m 49s)
• 14:45 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 (re)pooling @ 50%: Post clone repool (src)', diff saved to https://phabricator.wikimedia.org/P59495 and previous config saved to /var/cache/conftool/dbconfig/20240404-144536-arnaudb.json
• 14:45 arnaudb@cumin1002: dbctl commit (dc=all): 'db2213 (re)pooling @ 2%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59494 and previous config saved to /var/cache/conftool/dbconfig/20240404-144526-arnaudb.json
• 14:45 arnaudb@cumin1002: dbctl commit (dc=all): 'db2207 (re)pooling @ 2%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59493 and previous config saved to /var/cache/conftool/dbconfig/20240404-144511-arnaudb.json
• 14:41 arnaudb@cumin1002: dbctl commit (dc=all): 'db2107 (re)pooling @ 50%: Post clone repool (src)', diff saved to https://phabricator.wikimedia.org/P59492 and previous config saved to /var/cache/conftool/dbconfig/20240404-144111-arnaudb.json
• 14:41 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P59491 and previous config saved to /var/cache/conftool/dbconfig/20240404-144105-arnaudb.json
• 14:36 dreamyjazz@deploy1002: dreamyjazz: Continuing with sync
• 14:36 dreamyjazz@deploy1002: dreamyjazz: Backport for Remove sk translation of centralauth-rightslog-name (T361695) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 14:33 dreamyjazz@deploy1002: Started scap: Backport for Remove sk translation of centralauth-rightslog-name (T361695)
• 14:30 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 (re)pooling @ 30%: Post clone repool (src)', diff saved to https://phabricator.wikimedia.org/P59490 and previous config saved to /var/cache/conftool/dbconfig/20240404-143030-arnaudb.json
• 14:30 arnaudb@cumin1002: dbctl commit (dc=all): 'db2213 (re)pooling @ 1%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59489 and previous config saved to /var/cache/conftool/dbconfig/20240404-143020-arnaudb.json
• 14:30 arnaudb@cumin1002: dbctl commit (dc=all): 'db2207 (re)pooling @ 1%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59488 and previous config saved to /var/cache/conftool/dbconfig/20240404-143006-arnaudb.json
• 14:26 arnaudb@cumin1002: dbctl commit (dc=all): 'db2107 (re)pooling @ 30%: Post clone repool (src)', diff saved to https://phabricator.wikimedia.org/P59487 and previous config saved to /var/cache/conftool/dbconfig/20240404-142606-arnaudb.json
• 14:25 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P59486 and previous config saved to /var/cache/conftool/dbconfig/20240404-142558-arnaudb.json
• 14:22 logmsgbot: lucaswerkmeister-wmde@deploy1002 Finished scap: Backport for Enable wgVisualEditorAllowExternalLinkPaste at collabwiki (duration: 18m 43s)
• 14:15 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 (re)pooling @ 20%: Post clone repool (src)', diff saved to https://phabricator.wikimedia.org/P59485 and previous config saved to /var/cache/conftool/dbconfig/20240404-141517-arnaudb.json
• 14:11 arnaudb@cumin1002: dbctl commit (dc=all): 'db2107 (re)pooling @ 20%: Post clone repool (src)', diff saved to https://phabricator.wikimedia.org/P59484 and previous config saved to /var/cache/conftool/dbconfig/20240404-141100-arnaudb.json
• 14:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1170 (T360332)', diff saved to https://phabricator.wikimedia.org/P59483 and previous config saved to /var/cache/conftool/dbconfig/20240404-141051-arnaudb.json
• 14:09 logmsgbot: lucaswerkmeister-wmde@deploy1002 esanders and lucaswerkmeister-wmde: Continuing with sync
• 14:05 logmsgbot: lucaswerkmeister-wmde@deploy1002 esanders and lucaswerkmeister-wmde: Backport for Enable wgVisualEditorAllowExternalLinkPaste at collabwiki synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 14:03 logmsgbot: lucaswerkmeister-wmde@deploy1002 Started scap: Backport for Enable wgVisualEditorAllowExternalLinkPaste at collabwiki
• 14:01 logmsgbot: lucaswerkmeister-wmde@deploy1002 Finished scap: Backport for End EditCheck add-a-reference A/B test (T361727) (duration: 20m 05s)
• 14:00 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1193 (T355609)', diff saved to https://phabricator.wikimedia.org/P59482 and previous config saved to /var/cache/conftool/dbconfig/20240404-140050-marostegui.json
• 14:00 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1193.eqiad.wmnet with reason: Maintenance
• 14:00 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1193.eqiad.wmnet with reason: Maintenance
• 14:00 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192 (T355609)', diff saved to https://phabricator.wikimedia.org/P59481 and previous config saved to /var/cache/conftool/dbconfig/20240404-140027-marostegui.json
• 14:00 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 (re)pooling @ 10%: Post clone repool (src)', diff saved to https://phabricator.wikimedia.org/P59480 and previous config saved to /var/cache/conftool/dbconfig/20240404-140011-arnaudb.json
• 13:55 arnaudb@cumin1002: dbctl commit (dc=all): 'db2107 (re)pooling @ 10%: Post clone repool (src)', diff saved to https://phabricator.wikimedia.org/P59479 and previous config saved to /var/cache/conftool/dbconfig/20240404-135547-arnaudb.json
• 13:48 logmsgbot: lucaswerkmeister-wmde@deploy1002 lucaswerkmeister-wmde and esanders: Continuing with sync
• 13:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1170 (T360332)', diff saved to https://phabricator.wikimedia.org/P59478 and previous config saved to /var/cache/conftool/dbconfig/20240404-134607-arnaudb.json
• 13:46 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1170.eqiad.wmnet with reason: Maintenance
• 13:45 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1170.eqiad.wmnet with reason: Maintenance
• 13:45 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T360332)', diff saved to https://phabricator.wikimedia.org/P59477 and previous config saved to /var/cache/conftool/dbconfig/20240404-134544-arnaudb.json
• 13:45 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P59476 and previous config saved to /var/cache/conftool/dbconfig/20240404-134519-marostegui.json
• 13:43 logmsgbot: lucaswerkmeister-wmde@deploy1002 lucaswerkmeister-wmde and esanders: Backport for End EditCheck add-a-reference A/B test (T361727) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 13:41 logmsgbot: lucaswerkmeister-wmde@deploy1002 Started scap: Backport for End EditCheck add-a-reference A/B test (T361727)
• 13:40 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) Will create a clone of db2107.codfw.wmnet onto db2207.codfw.wmnet
• 13:39 logmsgbot: lucaswerkmeister-wmde@deploy1002 Finished scap: Backport for DiscussionTools: Remove no-op config (duration: 15m 10s)
• 13:30 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P59475 and previous config saved to /var/cache/conftool/dbconfig/20240404-133037-arnaudb.json
• 13:30 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P59474 and previous config saved to /var/cache/conftool/dbconfig/20240404-133012-marostegui.json
• 13:26 logmsgbot: lucaswerkmeister-wmde@deploy1002 lucaswerkmeister-wmde and esanders: Continuing with sync
• 13:26 logmsgbot: lucaswerkmeister-wmde@deploy1002 lucaswerkmeister-wmde and esanders: Backport for DiscussionTools: Remove no-op config synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 13:24 logmsgbot: lucaswerkmeister-wmde@deploy1002 Started scap: Backport for DiscussionTools: Remove no-op config
• 13:15 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P59473 and previous config saved to /var/cache/conftool/dbconfig/20240404-131529-arnaudb.json
• 13:15 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192 (T355609)', diff saved to https://phabricator.wikimedia.org/P59472 and previous config saved to /var/cache/conftool/dbconfig/20240404-131504-marostegui.json
• 13:04 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) Will create a clone of db2113.codfw.wmnet onto db2213.codfw.wmnet
• 13:00 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T360332)', diff saved to https://phabricator.wikimedia.org/P59471 and previous config saved to /var/cache/conftool/dbconfig/20240404-130022-arnaudb.json
• 12:42 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1206 (T356166)', diff saved to https://phabricator.wikimedia.org/P59470 and previous config saved to /var/cache/conftool/dbconfig/20240404-124257-marostegui.json
• 12:42 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1206.eqiad.wmnet with reason: Maintenance
• 12:42 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1206.eqiad.wmnet with reason: Maintenance
• 12:42 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1196 (T356166)', diff saved to https://phabricator.wikimedia.org/P59469 and previous config saved to /var/cache/conftool/dbconfig/20240404-124235-marostegui.json
• 12:36 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1158 (T360332)', diff saved to https://phabricator.wikimedia.org/P59468 and previous config saved to /var/cache/conftool/dbconfig/20240404-123645-arnaudb.json
• 12:36 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
• 12:36 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
• 12:36 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1158.eqiad.wmnet with reason: Maintenance
• 12:36 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1158.eqiad.wmnet with reason: Maintenance
• 12:27 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P59467 and previous config saved to /var/cache/conftool/dbconfig/20240404-122727-marostegui.json
• 12:25 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1192 (T355609)', diff saved to https://phabricator.wikimedia.org/P59466 and previous config saved to /var/cache/conftool/dbconfig/20240404-122557-marostegui.json
• 12:25 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1192.eqiad.wmnet with reason: Maintenance
• 12:25 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1192.eqiad.wmnet with reason: Maintenance
• 12:25 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178 (T355609)', diff saved to https://phabricator.wikimedia.org/P59465 and previous config saved to /var/cache/conftool/dbconfig/20240404-122535-marostegui.json
• 12:18 arnaudb@cumin1002: START - Cookbook sre.mysql.clone Will create a clone of db2113.codfw.wmnet onto db2213.codfw.wmnet
• 12:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Cloning db2113 in db2213 for T355422', diff saved to https://phabricator.wikimedia.org/P59463 and previous config saved to /var/cache/conftool/dbconfig/20240404-121722-arnaudb.json
• 12:16 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: provisionning db2213.codfw.wmnet - T355422
• 12:16 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: provisionning db2213.codfw.wmnet - T355422
• 12:16 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2113.codfw.wmnet with reason: provisionning db2213.codfw.wmnet - T355422
• 12:16 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2113.codfw.wmnet with reason: provisionning db2213.codfw.wmnet - T355422
• 12:12 arnaudb@cumin1002: START - Cookbook sre.mysql.clone Will create a clone of db2107.codfw.wmnet onto db2207.codfw.wmnet
• 12:12 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P59462 and previous config saved to /var/cache/conftool/dbconfig/20240404-121218-marostegui.json
• 12:11 aikochou@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
• 12:10 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P59461 and previous config saved to /var/cache/conftool/dbconfig/20240404-121027-marostegui.json
• 12:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Cloning db2107 in db2207 for T355422', diff saved to https://phabricator.wikimedia.org/P59460 and previous config saved to /var/cache/conftool/dbconfig/20240404-121008-arnaudb.json
• 12:08 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2207.codfw.wmnet with reason: provisionning db2207.codfw.wmnet - T355422
• 12:08 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2207.codfw.wmnet with reason: provisionning db2207.codfw.wmnet - T355422
• 12:08 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2107.codfw.wmnet with reason: provisionning db2207.codfw.wmnet - T355422
• 12:08 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2107.codfw.wmnet with reason: provisionning db2207.codfw.wmnet - T355422
• 11:57 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1196 (T356166)', diff saved to https://phabricator.wikimedia.org/P59459 and previous config saved to /var/cache/conftool/dbconfig/20240404-115709-marostegui.json
• 11:55 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P59458 and previous config saved to /var/cache/conftool/dbconfig/20240404-115520-marostegui.json
• 11:49 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.debug (exit_code=0) for Netbox circuit ID 108
• 11:49 ayounsi@cumin1002: START - Cookbook sre.network.debug for Netbox circuit ID 108
• 11:40 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178 (T355609)', diff saved to https://phabricator.wikimedia.org/P59457 and previous config saved to /var/cache/conftool/dbconfig/20240404-114012-marostegui.json
• 10:52 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1178 (T355609)', diff saved to https://phabricator.wikimedia.org/P59456 and previous config saved to /var/cache/conftool/dbconfig/20240404-105158-marostegui.json
• 10:51 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1178.eqiad.wmnet with reason: Maintenance
• 10:51 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1178.eqiad.wmnet with reason: Maintenance
• 10:51 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177 (T355609)', diff saved to https://phabricator.wikimedia.org/P59455 and previous config saved to /var/cache/conftool/dbconfig/20240404-105135-marostegui.json
• 10:36 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P59453 and previous config saved to /var/cache/conftool/dbconfig/20240404-103628-marostegui.json
• 10:34 jgiannelos@deploy1002: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
• 10:34 jgiannelos@deploy1002: helmfile [staging] START helmfile.d/services/mobileapps: apply
• 10:33 jgiannelos@deploy1002: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
• 10:33 jgiannelos@deploy1002: helmfile [staging] START helmfile.d/services/mobileapps: apply
• 10:21 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P59451 and previous config saved to /var/cache/conftool/dbconfig/20240404-102120-marostegui.json
• 10:10 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-role (exit_code=0) for role: mariadb::objectstash
• 10:06 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177 (T355609)', diff saved to https://phabricator.wikimedia.org/P59450 and previous config saved to /var/cache/conftool/dbconfig/20240404-100612-marostegui.json
• 10:05 jgiannelos@deploy1002: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
• 10:05 jgiannelos@deploy1002: helmfile [staging] START helmfile.d/services/mobileapps: apply
• 09:57 jmm@cumin2002: START - Cookbook sre.puppet.migrate-role for role: mariadb::objectstash
• 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-role (exit_code=0) for role: mariadb::misc
• 09:49 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2104.codfw.wmnet
• 09:49 marostegui@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 09:49 marostegui@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2104.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
• 09:48 marostegui@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2104.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
• 09:46 marostegui@cumin1002: START - Cookbook sre.dns.netbox
• 09:46 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1196 (T356166)', diff saved to https://phabricator.wikimedia.org/P59449 and previous config saved to /var/cache/conftool/dbconfig/20240404-094608-marostegui.json
• 09:46 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
• 09:45 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 16:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
• 09:45 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1196.eqiad.wmnet with reason: Maintenance
• 09:45 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1196.eqiad.wmnet with reason: Maintenance
• 09:45 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1186 (T356166)', diff saved to https://phabricator.wikimedia.org/P59448 and previous config saved to /var/cache/conftool/dbconfig/20240404-094536-marostegui.json
• 09:41 marostegui@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2104.codfw.wmnet
• 09:31 jmm@cumin2002: START - Cookbook sre.puppet.migrate-role for role: mariadb::misc
• 09:30 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P59446 and previous config saved to /var/cache/conftool/dbconfig/20240404-093028-marostegui.json
• 09:18 arnaudb@cumin1002: dbctl commit (dc=all): 'bump db2113 weight', diff saved to https://phabricator.wikimedia.org/P59445 and previous config saved to /var/cache/conftool/dbconfig/20240404-091858-arnaudb.json
• 09:17 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1177 (T355609)', diff saved to https://phabricator.wikimedia.org/P59444 and previous config saved to /var/cache/conftool/dbconfig/20240404-091732-marostegui.json
• 09:17 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1177.eqiad.wmnet with reason: Maintenance
• 09:17 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1177.eqiad.wmnet with reason: Maintenance
• 09:17 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172 (T355609)', diff saved to https://phabricator.wikimedia.org/P59443 and previous config saved to /var/cache/conftool/dbconfig/20240404-091709-marostegui.json
• 09:15 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P59442 and previous config saved to /var/cache/conftool/dbconfig/20240404-091521-marostegui.json
• 09:15 arnaudb@cumin1002: dbctl commit (dc=all): 'Promote db2123 to s5 primary T361789', diff saved to https://phabricator.wikimedia.org/P59441 and previous config saved to /var/cache/conftool/dbconfig/20240404-091512-arnaudb.json
• 09:14 arnaudb: Starting s5 codfw failover from db2113 to db2123 - T361789
• 09:11 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-role (exit_code=0) for role: mariadb::backup_source
• 09:02 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P59440 and previous config saved to /var/cache/conftool/dbconfig/20240404-090202-marostegui.json
• 09:00 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1186 (T356166)', diff saved to https://phabricator.wikimedia.org/P59439 and previous config saved to /var/cache/conftool/dbconfig/20240404-090007-marostegui.json
• 08:59 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1186 (T356166)', diff saved to https://phabricator.wikimedia.org/P59438 and previous config saved to /var/cache/conftool/dbconfig/20240404-085856-marostegui.json
• 08:58 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1186.eqiad.wmnet with reason: Maintenance
• 08:58 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1186.eqiad.wmnet with reason: Maintenance
• 08:58 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1169 (T356166)', diff saved to https://phabricator.wikimedia.org/P59437 and previous config saved to /var/cache/conftool/dbconfig/20240404-085834-marostegui.json
• 08:56 arnaudb@cumin1002: dbctl commit (dc=all): 'Set db2123 with weight 0 T361789', diff saved to https://phabricator.wikimedia.org/P59436 and previous config saved to /var/cache/conftool/dbconfig/20240404-085606-arnaudb.json
• 08:55 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 27 hosts with reason: Primary switchover s5 T361789
• 08:55 jmm@cumin2002: START - Cookbook sre.puppet.migrate-role for role: mariadb::backup_source
• 08:55 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on 27 hosts with reason: Primary switchover s5 T361789
• 08:46 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P59435 and previous config saved to /var/cache/conftool/dbconfig/20240404-084655-marostegui.json
• 08:43 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P59434 and previous config saved to /var/cache/conftool/dbconfig/20240404-084327-marostegui.json
• 08:31 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172 (T355609)', diff saved to https://phabricator.wikimedia.org/P59433 and previous config saved to /var/cache/conftool/dbconfig/20240404-083147-marostegui.json
• 08:28 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P59432 and previous config saved to /var/cache/conftool/dbconfig/20240404-082819-marostegui.json
• 08:25 arnaudb@cumin1002: dbctl commit (dc=all): 'bump db2107 weight', diff saved to https://phabricator.wikimedia.org/P59431 and previous config saved to /var/cache/conftool/dbconfig/20240404-082547-root.json
• 08:24 brouberol@deploy1002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 08:23 brouberol@deploy1002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 08:22 arnaudb@cumin1002: dbctl commit (dc=all): 'Promote db2204 to s2 primary T361682', diff saved to https://phabricator.wikimedia.org/P59430 and previous config saved to /var/cache/conftool/dbconfig/20240404-082200-arnaudb.json
• 08:04 arnaudb@cumin1002: dbctl commit (dc=all): 'Set db2204 with weight 0 T361682', diff saved to https://phabricator.wikimedia.org/P59427 and previous config saved to /var/cache/conftool/dbconfig/20240404-080408-arnaudb.json
• 08:02 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 29 hosts with reason: Primary switchover s2 T361682
• 08:01 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on 29 hosts with reason: Primary switchover s2 T361682
• 07:43 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1172 (T355609)', diff saved to https://phabricator.wikimedia.org/P59426 and previous config saved to /var/cache/conftool/dbconfig/20240404-074313-marostegui.json
• 07:43 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1172.eqiad.wmnet with reason: Maintenance
• 07:42 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1172.eqiad.wmnet with reason: Maintenance
• 07:36 marostegui@cumin1002: dbctl commit (dc=all): 'Remove db2104 from dbctl T361779', diff saved to https://phabricator.wikimedia.org/P59425 and previous config saved to /var/cache/conftool/dbconfig/20240404-073600-root.json
• 07:29 marostegui@cumin1002: dbctl commit (dc=all): 'db2126 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P59424 and previous config saved to /var/cache/conftool/dbconfig/20240404-072928-root.json
• 07:14 marostegui@cumin1002: dbctl commit (dc=all): 'db2126 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P59423 and previous config saved to /var/cache/conftool/dbconfig/20240404-071423-root.json
• 06:59 marostegui@cumin1002: dbctl commit (dc=all): 'db2126 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P59422 and previous config saved to /var/cache/conftool/dbconfig/20240404-065917-root.json
• 06:59 moritzm: installing util-linux security updates
• 06:58 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1171.eqiad.wmnet with reason: Maintenance
• 06:58 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1171.eqiad.wmnet with reason: Maintenance
• 06:58 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167 (T355609)', diff saved to https://phabricator.wikimedia.org/P59421 and previous config saved to /var/cache/conftool/dbconfig/20240404-065758-marostegui.json
• 06:44 marostegui@cumin1002: dbctl commit (dc=all): 'db2126 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P59420 and previous config saved to /var/cache/conftool/dbconfig/20240404-064411-root.json
• 06:42 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P59419 and previous config saved to /var/cache/conftool/dbconfig/20240404-064250-marostegui.json
• 06:29 marostegui@cumin1002: dbctl commit (dc=all): 'db2126 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P59418 and previous config saved to /var/cache/conftool/dbconfig/20240404-062905-root.json
• 06:27 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P59417 and previous config saved to /var/cache/conftool/dbconfig/20240404-062743-marostegui.json
• 06:14 marostegui@cumin1002: dbctl commit (dc=all): 'db2126 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P59416 and previous config saved to /var/cache/conftool/dbconfig/20240404-061400-root.json
• 06:12 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167 (T355609)', diff saved to https://phabricator.wikimedia.org/P59415 and previous config saved to /var/cache/conftool/dbconfig/20240404-061234-marostegui.json
• 06:05 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1169 (T356166)', diff saved to https://phabricator.wikimedia.org/P59414 and previous config saved to /var/cache/conftool/dbconfig/20240404-060524-marostegui.json
• 06:05 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1169.eqiad.wmnet with reason: Maintenance
• 06:05 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1169.eqiad.wmnet with reason: Maintenance
• 06:05 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1163 (T356166)', diff saved to https://phabricator.wikimedia.org/P59413 and previous config saved to /var/cache/conftool/dbconfig/20240404-060501-marostegui.json
• 06:01 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2126.codfw.wmnet with OS bookworm
• 05:58 marostegui@cumin1002: dbctl commit (dc=all): 'db2126 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P59412 and previous config saved to /var/cache/conftool/dbconfig/20240404-055854-root.json
• 05:49 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1163', diff saved to https://phabricator.wikimedia.org/P59411 and previous config saved to /var/cache/conftool/dbconfig/20240404-054953-marostegui.json
• 05:39 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2126.codfw.wmnet with reason: host reimage
• 05:36 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2126.codfw.wmnet with reason: host reimage
• 05:34 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1163', diff saved to https://phabricator.wikimedia.org/P59410 and previous config saved to /var/cache/conftool/dbconfig/20240404-053446-marostegui.json
• 05:23 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1167 (T355609)', diff saved to https://phabricator.wikimedia.org/P59409 and previous config saved to /var/cache/conftool/dbconfig/20240404-052338-marostegui.json
• 05:23 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
• 05:23 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 16:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
• 05:23 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1167.eqiad.wmnet with reason: Maintenance
• 05:23 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1167.eqiad.wmnet with reason: Maintenance
• 05:19 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1163 (T356166)', diff saved to https://phabricator.wikimedia.org/P59408 and previous config saved to /var/cache/conftool/dbconfig/20240404-051938-marostegui.json
• 05:19 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db2126.codfw.wmnet with OS bookworm
• 05:18 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2126 T361543', diff saved to https://phabricator.wikimedia.org/P59407 and previous config saved to /var/cache/conftool/dbconfig/20240404-051758-root.json
• 05:17 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1163 (T356166)', diff saved to https://phabricator.wikimedia.org/P59406 and previous config saved to /var/cache/conftool/dbconfig/20240404-051728-marostegui.json
• 05:17 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1163.eqiad.wmnet with reason: Maintenance
• 05:17 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1163.eqiad.wmnet with reason: Maintenance
• 02:48 TimStarling: ran maintain-views on clouddb1013-1019 (T355034)
• 02:29 eileen: civicrm upgraded from a0fb57d3 to 8c7cc208
• 02:28 eileen: config revision changed from 3ed18c47 to abccfdc0
• 02:11 eileen: config revision changed from 3ed18c47 to abccfdc0
• 01:07 TimStarling: on clouddb1020 running maintain-views --all-databases --replace-all --auto-depool (T355034)
• 00:56 TimStarling: on clouddb1021 ran maintain-views for all databases

2024-04-03

• 23:33 TimStarling: on clouddb1021 ran maintain-views for enwiki
• 22:35 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 22:34 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 22:34 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 22:34 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 22:26 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2217 (T360332)', diff saved to https://phabricator.wikimedia.org/P59405 and previous config saved to /var/cache/conftool/dbconfig/20240403-222610-arnaudb.json
• 22:11 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2217', diff saved to https://phabricator.wikimedia.org/P59404 and previous config saved to /var/cache/conftool/dbconfig/20240403-221103-arnaudb.json
• 22:09 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 22:09 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 22:09 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 22:06 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 22:06 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 22:06 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 22:06 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 22:06 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 22:05 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 22:00 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 22:00 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 22:00 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 21:59 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 21:59 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 21:59 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 21:58 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 21:58 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 21:58 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 21:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2217', diff saved to https://phabricator.wikimedia.org/P59403 and previous config saved to /var/cache/conftool/dbconfig/20240403-215555-arnaudb.json
• 21:40 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2217 (T360332)', diff saved to https://phabricator.wikimedia.org/P59402 and previous config saved to /var/cache/conftool/dbconfig/20240403-214048-arnaudb.json
• 21:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2217 (T360332)', diff saved to https://phabricator.wikimedia.org/P59401 and previous config saved to /var/cache/conftool/dbconfig/20240403-213825-arnaudb.json
• 21:38 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2217.codfw.wmnet with reason: Maintenance
• 21:38 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2217.codfw.wmnet with reason: Maintenance
• 21:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2214 (T360332)', diff saved to https://phabricator.wikimedia.org/P59400 and previous config saved to /var/cache/conftool/dbconfig/20240403-213802-arnaudb.json
• 21:22 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2214', diff saved to https://phabricator.wikimedia.org/P59399 and previous config saved to /var/cache/conftool/dbconfig/20240403-212255-arnaudb.json
• 21:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2214', diff saved to https://phabricator.wikimedia.org/P59398 and previous config saved to /var/cache/conftool/dbconfig/20240403-210747-arnaudb.json
• 21:02 jforrester@deploy1002: Finished scap: Backport for component: Add SandboxLink to Portuguese Wikiquote (T361447) (duration: 14m 18s)
• 20:52 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2214 (T360332)', diff saved to https://phabricator.wikimedia.org/P59397 and previous config saved to /var/cache/conftool/dbconfig/20240403-205240-arnaudb.json
• 20:51 jforrester@deploy1002: ederporto and jforrester: Continuing with sync
• 20:50 jforrester@deploy1002: ederporto and jforrester: Backport for component: Add SandboxLink to Portuguese Wikiquote (T361447) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:50 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 20:50 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 20:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2214 (T360332)', diff saved to https://phabricator.wikimedia.org/P59396 and previous config saved to /var/cache/conftool/dbconfig/20240403-205014-arnaudb.json
• 20:50 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2214.codfw.wmnet with reason: Maintenance
• 20:49 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2214.codfw.wmnet with reason: Maintenance
• 20:49 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2193 (T360332)', diff saved to https://phabricator.wikimedia.org/P59395 and previous config saved to /var/cache/conftool/dbconfig/20240403-204949-arnaudb.json
• 20:48 jforrester@deploy1002: Started scap: Backport for component: Add SandboxLink to Portuguese Wikiquote (T361447)
• 20:45 jforrester@deploy1002: Finished scap: Backport for Update the WikiLambda instrumentation to use core interaction events (T350497) (duration: 19m 03s)
• 20:34 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P59394 and previous config saved to /var/cache/conftool/dbconfig/20240403-203440-arnaudb.json
• 20:34 jforrester@deploy1002: sfaci and jforrester: Continuing with sync
• 20:28 jforrester@deploy1002: sfaci and jforrester: Backport for Update the WikiLambda instrumentation to use core interaction events (T350497) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:26 jforrester@deploy1002: Started scap: Backport for Update the WikiLambda instrumentation to use core interaction events (T350497)
• 20:23 jforrester@deploy1002: Finished scap: Backport for Centralize API calls in api.js mixin and fix error handling (T361598 T315432) (duration: 14m 58s)
• 20:19 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P59393 and previous config saved to /var/cache/conftool/dbconfig/20240403-201933-arnaudb.json
• 20:12 jforrester@deploy1002: jforrester: Continuing with sync
• 20:11 jforrester@deploy1002: jforrester: Backport for Centralize API calls in api.js mixin and fix error handling (T361598 T315432) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:10 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 20:10 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 20:08 jforrester@deploy1002: Started scap: Backport for Centralize API calls in api.js mixin and fix error handling (T361598 T315432)
• 20:04 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2193 (T360332)', diff saved to https://phabricator.wikimedia.org/P59392 and previous config saved to /var/cache/conftool/dbconfig/20240403-200425-arnaudb.json
• 20:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2193 (T360332)', diff saved to https://phabricator.wikimedia.org/P59391 and previous config saved to /var/cache/conftool/dbconfig/20240403-200201-arnaudb.json
• 20:01 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2193.codfw.wmnet with reason: Maintenance
• 20:01 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2193.codfw.wmnet with reason: Maintenance
• 20:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T360332)', diff saved to https://phabricator.wikimedia.org/P59390 and previous config saved to /var/cache/conftool/dbconfig/20240403-200137-arnaudb.json
• 20:00 mutante: stewards1001 - rebooting to switch from iptables to nftables
• 19:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P59388 and previous config saved to /var/cache/conftool/dbconfig/20240403-194630-arnaudb.json
• 19:38 mutante: stewards2001 - reboot to switch from iptables to nftables
• 19:31 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 19:31 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 19:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P59387 and previous config saved to /var/cache/conftool/dbconfig/20240403-193122-arnaudb.json
• 19:29 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 19:29 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 19:23 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 19:23 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 19:16 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 19:16 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 19:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T360332)', diff saved to https://phabricator.wikimedia.org/P59386 and previous config saved to /var/cache/conftool/dbconfig/20240403-191615-arnaudb.json
• 19:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2180 (T360332)', diff saved to https://phabricator.wikimedia.org/P59385 and previous config saved to /var/cache/conftool/dbconfig/20240403-191351-arnaudb.json
• 19:13 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2180.codfw.wmnet with reason: Maintenance
• 19:13 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2180.codfw.wmnet with reason: Maintenance
• 19:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2169 (T360332)', diff saved to https://phabricator.wikimedia.org/P59384 and previous config saved to /var/cache/conftool/dbconfig/20240403-191328-arnaudb.json
• 19:06 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 19:06 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 19:04 mvolz@deploy1002: helmfile [eqiad] DONE helmfile.d/services/zotero: apply
• 19:03 mvolz@deploy1002: helmfile [eqiad] START helmfile.d/services/zotero: apply
• 19:03 mvolz@deploy1002: helmfile [codfw] DONE helmfile.d/services/zotero: apply
• 19:02 mvolz@deploy1002: helmfile [codfw] START helmfile.d/services/zotero: apply
• 19:02 mvolz@deploy1002: helmfile [staging] DONE helmfile.d/services/zotero: apply
• 19:01 mvolz@deploy1002: helmfile [staging] START helmfile.d/services/zotero: apply
• 18:58 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2169', diff saved to https://phabricator.wikimedia.org/P59383 and previous config saved to /var/cache/conftool/dbconfig/20240403-185821-arnaudb.json
• 18:57 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 18:57 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 18:51 eileen: config revision changed from 821f145b to 3ed18c47
• 18:49 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 18:49 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 18:43 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2169', diff saved to https://phabricator.wikimedia.org/P59382 and previous config saved to /var/cache/conftool/dbconfig/20240403-184313-arnaudb.json
• 18:37 eevans@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts restbase[1019-1027].eqiad.wmnet
• 18:37 eevans@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 18:37 eevans@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: restbase[1019-1027].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - eevans@cumin1002"
• 18:35 eevans@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: restbase[1019-1027].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - eevans@cumin1002"
• 18:34 eevans@cumin1002: START - Cookbook sre.dns.netbox
• 18:28 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2169 (T360332)', diff saved to https://phabricator.wikimedia.org/P59381 and previous config saved to /var/cache/conftool/dbconfig/20240403-182806-arnaudb.json
• 18:25 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2169 (T360332)', diff saved to https://phabricator.wikimedia.org/P59380 and previous config saved to /var/cache/conftool/dbconfig/20240403-182543-arnaudb.json
• 18:25 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2169.codfw.wmnet with reason: Maintenance
• 18:25 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2169.codfw.wmnet with reason: Maintenance
• 18:25 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T360332)', diff saved to https://phabricator.wikimedia.org/P59379 and previous config saved to /var/cache/conftool/dbconfig/20240403-182520-arnaudb.json
• 18:13 logmsgbot: dreamyjazz Deployed security patch for T361479
• 18:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P59378 and previous config saved to /var/cache/conftool/dbconfig/20240403-181013-arnaudb.json
• 18:09 mvolz@deploy1002: helmfile [eqiad] DONE helmfile.d/services/citoid: apply
• 18:08 mvolz@deploy1002: helmfile [eqiad] START helmfile.d/services/citoid: apply
• 18:07 mvolz@deploy1002: helmfile [codfw] DONE helmfile.d/services/citoid: apply
• 18:07 eevans@cumin1002: START - Cookbook sre.hosts.decommission for hosts restbase[1019-1027].eqiad.wmnet
• 18:06 mvolz@deploy1002: helmfile [codfw] START helmfile.d/services/citoid: apply
• 18:05 mvolz@deploy1002: helmfile [staging] DONE helmfile.d/services/citoid: apply
• 18:04 mvolz@deploy1002: helmfile [staging] START helmfile.d/services/citoid: apply
• 18:03 marostegui@cumin1002: dbctl commit (dc=all): 'db1167 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P59377 and previous config saved to /var/cache/conftool/dbconfig/20240403-180323-root.json
• 17:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P59375 and previous config saved to /var/cache/conftool/dbconfig/20240403-175505-arnaudb.json
• 17:48 marostegui@cumin1002: dbctl commit (dc=all): 'db1167 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P59374 and previous config saved to /var/cache/conftool/dbconfig/20240403-174817-root.json
• 17:40 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T360332)', diff saved to https://phabricator.wikimedia.org/P59373 and previous config saved to /var/cache/conftool/dbconfig/20240403-173958-arnaudb.json
• 17:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2158 (T360332)', diff saved to https://phabricator.wikimedia.org/P59372 and previous config saved to /var/cache/conftool/dbconfig/20240403-173835-arnaudb.json
• 17:38 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance
• 17:38 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance
• 17:38 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2158.codfw.wmnet with reason: Maintenance
• 17:38 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2158.codfw.wmnet with reason: Maintenance
• 17:37 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2151 (T360332)', diff saved to https://phabricator.wikimedia.org/P59371 and previous config saved to /var/cache/conftool/dbconfig/20240403-173756-arnaudb.json
• 17:33 marostegui@cumin1002: dbctl commit (dc=all): 'db1167 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P59370 and previous config saved to /var/cache/conftool/dbconfig/20240403-173312-root.json
• 17:22 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2151', diff saved to https://phabricator.wikimedia.org/P59369 and previous config saved to /var/cache/conftool/dbconfig/20240403-172249-arnaudb.json
• 17:18 marostegui@cumin1002: dbctl commit (dc=all): 'db1167 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P59368 and previous config saved to /var/cache/conftool/dbconfig/20240403-171806-root.json
• 17:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2151', diff saved to https://phabricator.wikimedia.org/P59367 and previous config saved to /var/cache/conftool/dbconfig/20240403-170741-arnaudb.json
• 17:04 herron: performing rolling memory upgrades on prometheus100[56] T360687
• 17:03 marostegui@cumin1002: dbctl commit (dc=all): 'db1167 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P59366 and previous config saved to /var/cache/conftool/dbconfig/20240403-170300-root.json
• 16:59 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 16:59 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 16:54 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 16:54 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 16:52 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2151 (T360332)', diff saved to https://phabricator.wikimedia.org/P59365 and previous config saved to /var/cache/conftool/dbconfig/20240403-165234-arnaudb.json
• 16:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2151 (T360332)', diff saved to https://phabricator.wikimedia.org/P59364 and previous config saved to /var/cache/conftool/dbconfig/20240403-165011-arnaudb.json
• 16:50 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2151.codfw.wmnet with reason: Maintenance
• 16:49 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2151.codfw.wmnet with reason: Maintenance
• 16:49 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2124 (T360332)', diff saved to https://phabricator.wikimedia.org/P59363 and previous config saved to /var/cache/conftool/dbconfig/20240403-164948-arnaudb.json
• 16:47 marostegui@cumin1002: dbctl commit (dc=all): 'db1167 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P59362 and previous config saved to /var/cache/conftool/dbconfig/20240403-164754-root.json
• 16:38 jmm@cumin2002: END (PASS) - Cookbook sre.maps.roll-restart-reboot-master (exit_code=0) rolling restart_daemons on A:maps-master
• 16:36 jmm@cumin2002: START - Cookbook sre.maps.roll-restart-reboot-master rolling restart_daemons on A:maps-master
• 16:34 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2124', diff saved to https://phabricator.wikimedia.org/P59361 and previous config saved to /var/cache/conftool/dbconfig/20240403-163440-arnaudb.json
• 16:32 marostegui@cumin1002: dbctl commit (dc=all): 'db1167 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P59360 and previous config saved to /var/cache/conftool/dbconfig/20240403-163249-root.json
• 16:30 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1167 (T356166)', diff saved to https://phabricator.wikimedia.org/P59359 and previous config saved to /var/cache/conftool/dbconfig/20240403-163004-marostegui.json
• 16:29 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
• 16:29 jayme@deploy1002: Finished scap: (no justification provided) (duration: 03m 34s)
• 16:29 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 16:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
• 16:29 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1167.eqiad.wmnet with reason: Maintenance
• 16:29 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1167.eqiad.wmnet with reason: Maintenance
• 16:26 effie: pooling back mw-web-ro in eqiad
• 16:26 jayme@deploy1002: Started scap: (no justification provided)
• 16:26 jiji@cumin1002: conftool action : set/pooled=true; selector: dnsdisc=mw-web-ro,name=eqiad
• 16:19 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2124', diff saved to https://phabricator.wikimedia.org/P59358 and previous config saved to /var/cache/conftool/dbconfig/20240403-161933-arnaudb.json
• 16:14 jiji@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
• 16:12 jiji@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
• 16:08 akosiaris@deploy1002: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
• 16:07 akosiaris@deploy1002: helmfile [codfw] START helmfile.d/services/changeprop: apply
• 16:05 akosiaris@deploy1002: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
• 16:05 akosiaris@deploy1002: helmfile [eqiad] START helmfile.d/services/changeprop: apply
• 16:04 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2124 (T360332)', diff saved to https://phabricator.wikimedia.org/P59357 and previous config saved to /var/cache/conftool/dbconfig/20240403-160425-arnaudb.json
• 16:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2124 (T360332)', diff saved to https://phabricator.wikimedia.org/P59356 and previous config saved to /var/cache/conftool/dbconfig/20240403-160159-arnaudb.json
• 16:02 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2124.codfw.wmnet with reason: Maintenance
• 16:01 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2124.codfw.wmnet with reason: Maintenance
• 16:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2114 (T360332)', diff saved to https://phabricator.wikimedia.org/P59355 and previous config saved to /var/cache/conftool/dbconfig/20240403-160136-arnaudb.json
• 15:53 jiji@cumin1002: conftool action : set/pooled=false; selector: dnsdisc=mw-web-ro,name=eqiad
• 15:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2114', diff saved to https://phabricator.wikimedia.org/P59354 and previous config saved to /var/cache/conftool/dbconfig/20240403-154628-arnaudb.json
• 15:33 jiji@cumin1002: END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) status all services in all: None - None
• 15:33 jiji@cumin1002: START - Cookbook sre.discovery.datacenter status all services in all: None - None
• 15:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2114', diff saved to https://phabricator.wikimedia.org/P59353 and previous config saved to /var/cache/conftool/dbconfig/20240403-153121-arnaudb.json
• 15:22 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 15:22 Dreamy_Jazz: Starting MediaModeration scanning script again - It crashed due to the outage
• 15:22 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 15:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2114 (T360332)', diff saved to https://phabricator.wikimedia.org/P59352 and previous config saved to /var/cache/conftool/dbconfig/20240403-151614-arnaudb.json
• 15:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2114 (T360332)', diff saved to https://phabricator.wikimedia.org/P59351 and previous config saved to /var/cache/conftool/dbconfig/20240403-151349-arnaudb.json
• 15:13 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2114.codfw.wmnet with reason: Maintenance
• 15:13 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2114.codfw.wmnet with reason: Maintenance
• 15:13 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2097.codfw.wmnet with reason: Maintenance
• 15:12 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2097.codfw.wmnet with reason: Maintenance
• 15:12 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
• 15:12 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
• 15:12 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1231 (T360332)', diff saved to https://phabricator.wikimedia.org/P59350 and previous config saved to /var/cache/conftool/dbconfig/20240403-151233-arnaudb.json
• 15:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2098.codfw.wmnet with reason: restart of mysqld
• 15:03 jynus@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db2098.codfw.wmnet with reason: restart of mysqld
• 15:02 dreamyjazz@deploy1002: Finished scap: (no justification provided) (duration: 18m 54s)
• 15:01 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: sync
• 15:01 aborrero@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1040.eqiad.wmnet with OS bookworm
• 15:01 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/services/wikifeeds: sync
• 14:57 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P59349 and previous config saved to /var/cache/conftool/dbconfig/20240403-145725-arnaudb.json
• 14:44 dreamyjazz@deploy1002: Started scap: (no justification provided)
• 14:42 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P59347 and previous config saved to /var/cache/conftool/dbconfig/20240403-144217-arnaudb.json
• 14:34 aborrero@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1040.eqiad.wmnet with reason: host reimage
• 14:31 aborrero@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1040.eqiad.wmnet with reason: host reimage
• 14:27 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
• 14:27 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1231 (T360332)', diff saved to https://phabricator.wikimedia.org/P59346 and previous config saved to /var/cache/conftool/dbconfig/20240403-142709-arnaudb.json
• 14:27 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/admin 'apply'.
• 14:26 hnowlan@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
• 14:26 hnowlan@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
• 14:24 aborrero@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1040
• 14:24 aborrero@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1040
• 14:21 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
• 14:21 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
• 14:21 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1226 (T356166)', diff saved to https://phabricator.wikimedia.org/P59345 and previous config saved to /var/cache/conftool/dbconfig/20240403-142142-marostegui.json
• 14:17 jmm@cumin2002: END (PASS) - Cookbook sre.maps.roll-restart-reboot (exit_code=0) rolling restart_daemons on A:maps-replica-eqiad
• 14:16 aborrero@cumin1002: START - Cookbook sre.hosts.reimage for host cloudvirt1040.eqiad.wmnet with OS bookworm
• 14:11 jmm@cumin2002: START - Cookbook sre.maps.roll-restart-reboot rolling restart_daemons on A:maps-replica-eqiad
• 14:11 jmm@cumin2002: END (PASS) - Cookbook sre.maps.roll-restart-reboot (exit_code=0) rolling restart_daemons on A:maps-replica-codfw
• 14:09 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 14:09 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 14:08 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 2527
• 14:07 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 2527
• 14:06 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P59344 and previous config saved to /var/cache/conftool/dbconfig/20240403-140634-marostegui.json
• 14:06 jmm@cumin2002: START - Cookbook sre.maps.roll-restart-reboot rolling restart_daemons on A:maps-replica-codfw
• 13:51 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P59343 and previous config saved to /var/cache/conftool/dbconfig/20240403-135126-marostegui.json
• 13:41 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1231 (T360332)', diff saved to https://phabricator.wikimedia.org/P59342 and previous config saved to /var/cache/conftool/dbconfig/20240403-134136-arnaudb.json
• 13:41 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1231.eqiad.wmnet with reason: Maintenance
• 13:41 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1231.eqiad.wmnet with reason: Maintenance
• 13:40 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1225.eqiad.wmnet with reason: Maintenance
• 13:40 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1225.eqiad.wmnet with reason: Maintenance
• 13:40 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1224 (T360332)', diff saved to https://phabricator.wikimedia.org/P59341 and previous config saved to /var/cache/conftool/dbconfig/20240403-134044-arnaudb.json
• 13:36 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1226 (T356166)', diff saved to https://phabricator.wikimedia.org/P59340 and previous config saved to /var/cache/conftool/dbconfig/20240403-133619-marostegui.json
• 13:32 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1226 (T356166)', diff saved to https://phabricator.wikimedia.org/P59339 and previous config saved to /var/cache/conftool/dbconfig/20240403-133200-marostegui.json
• 13:31 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1226.eqiad.wmnet with reason: Maintenance
• 13:31 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1226.eqiad.wmnet with reason: Maintenance
• 13:31 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1216.eqiad.wmnet with reason: Maintenance
• 13:31 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1216.eqiad.wmnet with reason: Maintenance
• 13:16 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P59336 and previous config saved to /var/cache/conftool/dbconfig/20240403-131606-marostegui.json
• 13:15 moritzm: installing tiff security updates
• 13:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1224', diff saved to https://phabricator.wikimedia.org/P59335 and previous config saved to /var/cache/conftool/dbconfig/20240403-131029-arnaudb.json
• 13:00 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P59334 and previous config saved to /var/cache/conftool/dbconfig/20240403-130058-marostegui.json
• 12:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1224 (T360332)', diff saved to https://phabricator.wikimedia.org/P59333 and previous config saved to /var/cache/conftool/dbconfig/20240403-125521-arnaudb.json
• 12:45 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1214 (T356166)', diff saved to https://phabricator.wikimedia.org/P59332 and previous config saved to /var/cache/conftool/dbconfig/20240403-124550-marostegui.json
• 12:42 hashar: Upgrading CI Jenkins # T360759
• 12:34 aborrero@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1039.eqiad.wmnet with OS bookworm
• 12:33 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1214 (T356166)', diff saved to https://phabricator.wikimedia.org/P59330 and previous config saved to /var/cache/conftool/dbconfig/20240403-123329-marostegui.json
• 12:33 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1214.eqiad.wmnet with reason: Maintenance
• 12:33 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1214.eqiad.wmnet with reason: Maintenance
• 12:33 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1211 (T356166)', diff saved to https://phabricator.wikimedia.org/P59329 and previous config saved to /var/cache/conftool/dbconfig/20240403-123306-marostegui.json
• 12:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1224 (T360332)', diff saved to https://phabricator.wikimedia.org/P59328 and previous config saved to /var/cache/conftool/dbconfig/20240403-123156-arnaudb.json
• 12:31 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1224.eqiad.wmnet with reason: Maintenance
• 12:31 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1224.eqiad.wmnet with reason: Maintenance
• 12:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T360332)', diff saved to https://phabricator.wikimedia.org/P59327 and previous config saved to /var/cache/conftool/dbconfig/20240403-123133-arnaudb.json
• 12:17 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P59326 and previous config saved to /var/cache/conftool/dbconfig/20240403-121759-marostegui.json
• 12:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P59325 and previous config saved to /var/cache/conftool/dbconfig/20240403-121626-arnaudb.json
• 12:11 aborrero@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1039.eqiad.wmnet with reason: host reimage
• 12:08 aborrero@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1039.eqiad.wmnet with reason: host reimage
• 12:07 aborrero@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1039
• 12:02 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P59324 and previous config saved to /var/cache/conftool/dbconfig/20240403-120251-marostegui.json
• 12:02 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 12:02 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 12:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P59323 and previous config saved to /var/cache/conftool/dbconfig/20240403-120118-arnaudb.json
• 11:58 aborrero@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1039
• 11:52 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 11:52 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 11:50 aborrero@cumin1002: START - Cookbook sre.hosts.reimage for host cloudvirt1039.eqiad.wmnet with OS bookworm
• 11:47 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1211 (T356166)', diff saved to https://phabricator.wikimedia.org/P59322 and previous config saved to /var/cache/conftool/dbconfig/20240403-114743-marostegui.json
• 11:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T360332)', diff saved to https://phabricator.wikimedia.org/P59321 and previous config saved to /var/cache/conftool/dbconfig/20240403-114611-arnaudb.json
• 11:45 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1211 (T356166)', diff saved to https://phabricator.wikimedia.org/P59320 and previous config saved to /var/cache/conftool/dbconfig/20240403-114525-marostegui.json
• 11:45 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1211.eqiad.wmnet with reason: Maintenance
• 11:45 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1211.eqiad.wmnet with reason: Maintenance
• 11:45 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1203 (T356166)', diff saved to https://phabricator.wikimedia.org/P59319 and previous config saved to /var/cache/conftool/dbconfig/20240403-114502-marostegui.json
• 11:43 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1201 (T360332)', diff saved to https://phabricator.wikimedia.org/P59318 and previous config saved to /var/cache/conftool/dbconfig/20240403-114350-arnaudb.json
• 11:43 moritzm: installing imagemagick security updates
• 11:43 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1201.eqiad.wmnet with reason: Maintenance
• 11:43 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1201.eqiad.wmnet with reason: Maintenance
• 11:43 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T360332)', diff saved to https://phabricator.wikimedia.org/P59317 and previous config saved to /var/cache/conftool/dbconfig/20240403-114327-arnaudb.json
• 11:33 aborrero@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1038.eqiad.wmnet with OS bookworm
• 11:30 mvolz@deploy1002: helmfile [eqiad] DONE helmfile.d/services/citoid: apply
• 11:29 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P59315 and previous config saved to /var/cache/conftool/dbconfig/20240403-112955-marostegui.json
• 11:29 mvolz@deploy1002: helmfile [eqiad] START helmfile.d/services/citoid: apply
• 11:28 mvolz@deploy1002: helmfile [codfw] DONE helmfile.d/services/citoid: apply
• 11:28 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P59314 and previous config saved to /var/cache/conftool/dbconfig/20240403-112819-arnaudb.json
• 11:28 mvolz@deploy1002: helmfile [codfw] START helmfile.d/services/citoid: apply
• 11:27 mvolz@deploy1002: helmfile [staging] DONE helmfile.d/services/citoid: apply
• 11:27 mvolz@deploy1002: helmfile [staging] START helmfile.d/services/citoid: apply
• 11:25 jgiannelos@deploy1002: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
• 11:24 jgiannelos@deploy1002: helmfile [staging] START helmfile.d/services/mobileapps: apply
• 11:23 jgiannelos@deploy1002: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
• 11:23 jgiannelos@deploy1002: helmfile [staging] START helmfile.d/services/mobileapps: apply
• 11:16 aborrero@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1038
• 11:16 aborrero@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1038
• 11:16 mvolz@deploy1002: helmfile [eqiad] DONE helmfile.d/services/zotero: apply
• 11:15 mvolz@deploy1002: helmfile [eqiad] START helmfile.d/services/zotero: apply
• 11:14 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P59313 and previous config saved to /var/cache/conftool/dbconfig/20240403-111447-marostegui.json
• 11:14 mvolz@deploy1002: helmfile [codfw] DONE helmfile.d/services/zotero: apply
• 11:13 mvolz@deploy1002: helmfile [codfw] START helmfile.d/services/zotero: apply
• 11:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P59312 and previous config saved to /var/cache/conftool/dbconfig/20240403-111312-arnaudb.json
• 11:11 mvolz@deploy1002: helmfile [staging] DONE helmfile.d/services/zotero: apply
• 11:11 mvolz@deploy1002: helmfile [staging] START helmfile.d/services/zotero: apply
• 11:10 fab@deploy1002: Finished deploy [airflow-dags/research@75163c7]: (no justification provided) (duration: 00m 32s)
• 11:09 fab@deploy1002: Started deploy [airflow-dags/research@75163c7]: (no justification provided)
• 11:08 mvolz@deploy1002: helmfile [staging] DONE helmfile.d/services/zotero: apply
• 11:08 mvolz@deploy1002: helmfile [staging] START helmfile.d/services/zotero: apply
• 11:07 aborrero@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1038.eqiad.wmnet with reason: host reimage
• 11:05 mvolz@deploy1002: helmfile [staging] DONE helmfile.d/services/zotero: apply
• 11:05 mvolz@deploy1002: helmfile [staging] START helmfile.d/services/zotero: apply
• 11:04 aborrero@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1038.eqiad.wmnet with reason: host reimage
• 10:59 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1203 (T356166)', diff saved to https://phabricator.wikimedia.org/P59311 and previous config saved to /var/cache/conftool/dbconfig/20240403-105940-marostegui.json
• 10:58 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T360332)', diff saved to https://phabricator.wikimedia.org/P59310 and previous config saved to /var/cache/conftool/dbconfig/20240403-105804-arnaudb.json
• 10:57 jgiannelos@deploy1002: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
• 10:57 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1203 (T356166)', diff saved to https://phabricator.wikimedia.org/P59309 and previous config saved to /var/cache/conftool/dbconfig/20240403-105722-marostegui.json
• 10:57 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1203.eqiad.wmnet with reason: Maintenance
• 10:57 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1203.eqiad.wmnet with reason: Maintenance
• 10:57 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193 (T356166)', diff saved to https://phabricator.wikimedia.org/P59308 and previous config saved to /var/cache/conftool/dbconfig/20240403-105659-marostegui.json
• 10:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1187 (T360332)', diff saved to https://phabricator.wikimedia.org/P59307 and previous config saved to /var/cache/conftool/dbconfig/20240403-105545-arnaudb.json
• 10:55 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1187.eqiad.wmnet with reason: Maintenance
• 10:55 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1187.eqiad.wmnet with reason: Maintenance
• 10:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T360332)', diff saved to https://phabricator.wikimedia.org/P59306 and previous config saved to /var/cache/conftool/dbconfig/20240403-105522-arnaudb.json
• 10:47 jgiannelos@deploy1002: helmfile [staging] START helmfile.d/services/mobileapps: apply
• 10:47 aborrero@cumin1002: START - Cookbook sre.hosts.reimage for host cloudvirt1038.eqiad.wmnet with OS bookworm
• 10:41 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193', diff saved to https://phabricator.wikimedia.org/P59305 and previous config saved to /var/cache/conftool/dbconfig/20240403-104152-marostegui.json
• 10:40 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P59304 and previous config saved to /var/cache/conftool/dbconfig/20240403-104014-arnaudb.json
• 10:29 jgiannelos@deploy1002: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
• 10:29 jgiannelos@deploy1002: helmfile [staging] START helmfile.d/services/mobileapps: apply
• 10:26 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193', diff saved to https://phabricator.wikimedia.org/P59303 and previous config saved to /var/cache/conftool/dbconfig/20240403-102644-marostegui.json
• 10:25 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P59302 and previous config saved to /var/cache/conftool/dbconfig/20240403-102507-arnaudb.json
• 10:20 logmsgbot: dreamyjazz Deployed security patch for T361293
• 10:17 aborrero@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1037.eqiad.wmnet with OS bookworm
• 10:14 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 10:14 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 10:11 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193 (T356166)', diff saved to https://phabricator.wikimedia.org/P59301 and previous config saved to /var/cache/conftool/dbconfig/20240403-101137-marostegui.json
• 10:10 marostegui: Restart sanitarium db1154 T361673
• 10:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T360332)', diff saved to https://phabricator.wikimedia.org/P59300 and previous config saved to /var/cache/conftool/dbconfig/20240403-100959-arnaudb.json
• 10:09 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1193 (T356166)', diff saved to https://phabricator.wikimedia.org/P59299 and previous config saved to /var/cache/conftool/dbconfig/20240403-100919-marostegui.json
• 10:09 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1193.eqiad.wmnet with reason: Maintenance
• 10:09 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1193.eqiad.wmnet with reason: Maintenance
• 10:09 moritzm: installing util-linux security updates
• 10:08 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192 (T356166)', diff saved to https://phabricator.wikimedia.org/P59298 and previous config saved to /var/cache/conftool/dbconfig/20240403-100857-marostegui.json
• 10:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1180 (T360332)', diff saved to https://phabricator.wikimedia.org/P59297 and previous config saved to /var/cache/conftool/dbconfig/20240403-100735-arnaudb.json
• 10:07 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1180.eqiad.wmnet with reason: Maintenance
• 10:07 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1180.eqiad.wmnet with reason: Maintenance
• 10:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T360332)', diff saved to https://phabricator.wikimedia.org/P59296 and previous config saved to /var/cache/conftool/dbconfig/20240403-100712-arnaudb.json
• 10:06 logmsgbot: dreamyjazz Deployed security patch for T361293
• 09:53 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P59295 and previous config saved to /var/cache/conftool/dbconfig/20240403-095349-marostegui.json
• 09:52 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P59294 and previous config saved to /var/cache/conftool/dbconfig/20240403-095204-arnaudb.json
• 09:48 aborrero@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1037.eqiad.wmnet with reason: host reimage
• 09:45 aborrero@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1037.eqiad.wmnet with reason: host reimage
• 09:45 Dreamy_Jazz: Doing security deploy for T361293
• 09:43 aborrero@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1037
• 09:42 marostegui@cumin1002: dbctl commit (dc=all): 'db2125 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P59293 and previous config saved to /var/cache/conftool/dbconfig/20240403-094241-root.json
• 09:42 aborrero@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1037
• 09:39 godog: roll-restart prometheus/k8s in codfw/eqiad to apply new retention settings - T360537
• 09:38 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P59292 and previous config saved to /var/cache/conftool/dbconfig/20240403-093842-marostegui.json
• 09:36 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P59291 and previous config saved to /var/cache/conftool/dbconfig/20240403-093657-arnaudb.json
• 09:31 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 09:31 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 09:27 marostegui: Restart sanitarium db1155 T361673
• 09:27 marostegui@cumin1002: dbctl commit (dc=all): 'db2125 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P59290 and previous config saved to /var/cache/conftool/dbconfig/20240403-092735-root.json
• 09:27 aborrero@cumin1002: START - Cookbook sre.hosts.reimage for host cloudvirt1037.eqiad.wmnet with OS bookworm
• 09:24 fabfur@cumin1002: conftool action : set/pooled=yes; selector: name=cp3067.esams.wmnet
• 09:23 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192 (T356166)', diff saved to https://phabricator.wikimedia.org/P59289 and previous config saved to /var/cache/conftool/dbconfig/20240403-092334-marostegui.json
• 09:21 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T360332)', diff saved to https://phabricator.wikimedia.org/P59288 and previous config saved to /var/cache/conftool/dbconfig/20240403-092149-arnaudb.json
• 09:21 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1192 (T356166)', diff saved to https://phabricator.wikimedia.org/P59287 and previous config saved to /var/cache/conftool/dbconfig/20240403-092116-marostegui.json
• 09:21 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1192.eqiad.wmnet with reason: Maintenance
• 09:20 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1192.eqiad.wmnet with reason: Maintenance
• 09:20 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178 (T356166)', diff saved to https://phabricator.wikimedia.org/P59286 and previous config saved to /var/cache/conftool/dbconfig/20240403-092053-marostegui.json
• 09:19 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3067.esams.wmnet with OS bullseye
• 09:19 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1168 (T360332)', diff saved to https://phabricator.wikimedia.org/P59285 and previous config saved to /var/cache/conftool/dbconfig/20240403-091929-arnaudb.json
• 09:19 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1168.eqiad.wmnet with reason: Maintenance
• 09:19 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1168.eqiad.wmnet with reason: Maintenance
• 09:19 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T360332)', diff saved to https://phabricator.wikimedia.org/P59284 and previous config saved to /var/cache/conftool/dbconfig/20240403-091906-arnaudb.json
• 09:19 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 09:18 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 09:12 marostegui@cumin1002: dbctl commit (dc=all): 'db2125 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P59283 and previous config saved to /var/cache/conftool/dbconfig/20240403-091229-root.json
• 09:05 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P59282 and previous config saved to /var/cache/conftool/dbconfig/20240403-090545-marostegui.json
• 09:04 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 09:03 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P59281 and previous config saved to /var/cache/conftool/dbconfig/20240403-090358-arnaudb.json
• 09:03 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 09:00 slyngs: Upgraded Bitu / idm.wikimedia.org to version 0.0.6-2
• 08:57 marostegui@cumin1002: dbctl commit (dc=all): 'db2125 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P59280 and previous config saved to /var/cache/conftool/dbconfig/20240403-085723-root.json
• 08:55 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3067.esams.wmnet with reason: host reimage
• 08:52 fabfur@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3067.esams.wmnet with reason: host reimage
• 08:50 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P59279 and previous config saved to /var/cache/conftool/dbconfig/20240403-085037-marostegui.json
• 08:48 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P59278 and previous config saved to /var/cache/conftool/dbconfig/20240403-084851-arnaudb.json
• 08:42 marostegui@cumin1002: dbctl commit (dc=all): 'db2125 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P59276 and previous config saved to /var/cache/conftool/dbconfig/20240403-084218-root.json
• 08:36 marostegui: stop sanitarium codfw hosts T361673
• 08:35 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178 (T356166)', diff saved to https://phabricator.wikimedia.org/P59275 and previous config saved to /var/cache/conftool/dbconfig/20240403-083530-marostegui.json
• 08:33 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T360332)', diff saved to https://phabricator.wikimedia.org/P59274 and previous config saved to /var/cache/conftool/dbconfig/20240403-083343-arnaudb.json
• 08:33 jnuche@deploy1002: Synchronized php: group1 wikis to 1.42.0-wmf.25 refs T360157 (duration: 13m 00s)
• 08:33 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1178 (T356166)', diff saved to https://phabricator.wikimedia.org/P59273 and previous config saved to /var/cache/conftool/dbconfig/20240403-083313-marostegui.json
• 08:33 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1178.eqiad.wmnet with reason: Maintenance
• 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts puppetmaster1002.eqiad.wmnet
• 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: puppetmaster1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002"
• 08:32 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1178.eqiad.wmnet with reason: Maintenance
• 08:32 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177 (T356166)', diff saved to https://phabricator.wikimedia.org/P59272 and previous config saved to /var/cache/conftool/dbconfig/20240403-083249-marostegui.json
• 08:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1165 (T360332)', diff saved to https://phabricator.wikimedia.org/P59271 and previous config saved to /var/cache/conftool/dbconfig/20240403-083123-arnaudb.json
• 08:31 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
• 08:31 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
• 08:30 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1165.eqiad.wmnet with reason: Maintenance
• 08:30 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1165.eqiad.wmnet with reason: Maintenance
• 08:29 fabfur@cumin1002: START - Cookbook sre.hosts.reimage for host cp3067.esams.wmnet with OS bullseye
• 08:28 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: puppetmaster1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002"
• 08:27 marostegui@cumin1002: dbctl commit (dc=all): 'db2125 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P59270 and previous config saved to /var/cache/conftool/dbconfig/20240403-082712-root.json
• 08:27 fabfur@cumin1002: conftool action : set/pooled=no; selector: name=cp3067.esams.wmnet
• 08:24 jmm@cumin2002: START - Cookbook sre.dns.netbox
• 08:24 fabfur: depool cp3067 for reimage (T360430)
• 08:09 arnaudb@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2100.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
• 08:02 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P59267 and previous config saved to /var/cache/conftool/dbconfig/20240403-080235-marostegui.json
• 08:01 arnaudb@cumin1002: START - Cookbook sre.dns.netbox
• 07:58 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2125.codfw.wmnet with OS bookworm
• 07:55 marostegui@cumin1002: dbctl commit (dc=all): 'db2148 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P59266 and previous config saved to /var/cache/conftool/dbconfig/20240403-075510-root.json
• 07:53 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2202.codfw.wmnet with OS bookworm
• 07:47 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177 (T356166)', diff saved to https://phabricator.wikimedia.org/P59265 and previous config saved to /var/cache/conftool/dbconfig/20240403-074727-marostegui.json
• 07:45 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1177 (T356166)', diff saved to https://phabricator.wikimedia.org/P59264 and previous config saved to /var/cache/conftool/dbconfig/20240403-074509-marostegui.json
• 07:45 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1177.eqiad.wmnet with reason: Maintenance
• 07:44 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1177.eqiad.wmnet with reason: Maintenance
• 07:44 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172 (T356166)', diff saved to https://phabricator.wikimedia.org/P59263 and previous config saved to /var/cache/conftool/dbconfig/20240403-074446-marostegui.json
• 07:40 marostegui@cumin1002: dbctl commit (dc=all): 'db2148 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P59262 and previous config saved to /var/cache/conftool/dbconfig/20240403-074004-root.json
• 07:35 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2125.codfw.wmnet with reason: host reimage
• 07:32 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2202.codfw.wmnet with reason: host reimage
• 07:29 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P59261 and previous config saved to /var/cache/conftool/dbconfig/20240403-072938-marostegui.json
• 07:28 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2125.codfw.wmnet with reason: host reimage
• 07:27 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2202.codfw.wmnet with reason: host reimage
• 07:24 marostegui@cumin1002: dbctl commit (dc=all): 'db2148 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P59260 and previous config saved to /var/cache/conftool/dbconfig/20240403-072459-root.json
• 07:18 arnaudb@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2100.codfw.wmnet
• 07:17 marostegui@cumin1002: dbctl commit (dc=all): 'db1222 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P59259 and previous config saved to /var/cache/conftool/dbconfig/20240403-071718-root.json
• 07:14 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P59258 and previous config saved to /var/cache/conftool/dbconfig/20240403-071431-marostegui.json
• 07:11 arnaudb@cumin1002: START - Cookbook sre.hosts.reimage for host db2202.codfw.wmnet with OS bookworm
• 07:11 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db2125.codfw.wmnet with OS bookworm
• 07:09 marostegui@cumin1002: dbctl commit (dc=all): 'db2148 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P59257 and previous config saved to /var/cache/conftool/dbconfig/20240403-070953-root.json
• 07:09 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2125 T361543', diff saved to https://phabricator.wikimedia.org/P59256 and previous config saved to /var/cache/conftool/dbconfig/20240403-070946-root.json
• 07:02 marostegui@cumin1002: dbctl commit (dc=all): 'db1222 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P59255 and previous config saved to /var/cache/conftool/dbconfig/20240403-070212-root.json
• 06:59 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172 (T356166)', diff saved to https://phabricator.wikimedia.org/P59254 and previous config saved to /var/cache/conftool/dbconfig/20240403-065923-marostegui.json
• 06:57 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1172 (T356166)', diff saved to https://phabricator.wikimedia.org/P59253 and previous config saved to /var/cache/conftool/dbconfig/20240403-065706-marostegui.json
• 06:56 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1172.eqiad.wmnet with reason: Maintenance
• 06:56 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1172.eqiad.wmnet with reason: Maintenance
• 06:56 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1171.eqiad.wmnet with reason: Maintenance
• 06:56 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1171.eqiad.wmnet with reason: Maintenance
• 06:56 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167 (T356166)', diff saved to https://phabricator.wikimedia.org/P59252 and previous config saved to /var/cache/conftool/dbconfig/20240403-065617-marostegui.json
• 06:54 marostegui@cumin1002: dbctl commit (dc=all): 'db2148 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P59251 and previous config saved to /var/cache/conftool/dbconfig/20240403-065447-root.json
• 06:47 marostegui@cumin1002: dbctl commit (dc=all): 'db1222 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P59250 and previous config saved to /var/cache/conftool/dbconfig/20240403-064704-root.json
• 06:41 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P59249 and previous config saved to /var/cache/conftool/dbconfig/20240403-064110-marostegui.json
• 06:39 marostegui@cumin1002: dbctl commit (dc=all): 'db2148 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P59248 and previous config saved to /var/cache/conftool/dbconfig/20240403-063941-root.json
• 06:31 marostegui@cumin1002: dbctl commit (dc=all): 'db1222 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P59247 and previous config saved to /var/cache/conftool/dbconfig/20240403-063159-root.json
• 06:26 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P59246 and previous config saved to /var/cache/conftool/dbconfig/20240403-062602-marostegui.json
• 06:25 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2148.codfw.wmnet with OS bookworm
• 06:24 marostegui@cumin1002: dbctl commit (dc=all): 'db2148 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P59245 and previous config saved to /var/cache/conftool/dbconfig/20240403-062436-root.json
• 06:23 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 06:23 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 06:16 marostegui@cumin1002: dbctl commit (dc=all): 'db1222 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P59244 and previous config saved to /var/cache/conftool/dbconfig/20240403-061653-root.json
• 06:13 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 06:13 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 06:10 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167 (T356166)', diff saved to https://phabricator.wikimedia.org/P59243 and previous config saved to /var/cache/conftool/dbconfig/20240403-061055-marostegui.json
• 06:04 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2148.codfw.wmnet with reason: host reimage
• 06:01 marostegui@cumin1002: dbctl commit (dc=all): 'db1222 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P59242 and previous config saved to /var/cache/conftool/dbconfig/20240403-060147-root.json
• 06:01 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2148.codfw.wmnet with reason: host reimage
• 05:51 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 05:50 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 05:49 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1222.eqiad.wmnet with OS bookworm
• 05:46 marostegui@cumin1002: dbctl commit (dc=all): 'db1222 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P59241 and previous config saved to /var/cache/conftool/dbconfig/20240403-054641-root.json
• 05:44 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db2148.codfw.wmnet with OS bookworm
• 05:43 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2148 T361543', diff saved to https://phabricator.wikimedia.org/P59240 and previous config saved to /var/cache/conftool/dbconfig/20240403-054310-root.json
• 05:28 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1222.eqiad.wmnet with reason: host reimage
• 05:25 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1222.eqiad.wmnet with reason: host reimage
• 05:13 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db1222.eqiad.wmnet with OS bookworm
• 05:11 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1222 T361543', diff saved to https://phabricator.wikimedia.org/P59239 and previous config saved to /var/cache/conftool/dbconfig/20240403-051149-root.json
• 05:10 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1167 (T356166)', diff saved to https://phabricator.wikimedia.org/P59238 and previous config saved to /var/cache/conftool/dbconfig/20240403-051029-marostegui.json
• 05:10 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
• 05:10 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 16:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
• 05:10 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1167.eqiad.wmnet with reason: Maintenance
• 05:09 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1167.eqiad.wmnet with reason: Maintenance
• 01:58 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 01:58 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 01:51 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 01:51 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 01:40 cwhite@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host logging-hd2002.codfw.wmnet with OS bookworm
• 01:19 cwhite@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logging-hd2002.codfw.wmnet with reason: host reimage
• 01:15 cwhite@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on logging-hd2002.codfw.wmnet with reason: host reimage
• 01:06 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 01:06 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 01:00 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 01:00 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 00:46 cwhite@cumin2002: START - Cookbook sre.hosts.reimage for host logging-hd2002.codfw.wmnet with OS bookworm
• 00:44 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 00:44 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 00:43 cwhite@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host logging-hd2002.codfw.wmnet with OS bookworm
• 00:37 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 00:36 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 00:30 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 00:30 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 00:25 cwhite@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host logging-hd2003.codfw.wmnet with OS bookworm
• 00:25 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 00:25 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 00:23 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 00:23 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 00:17 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 00:17 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 00:13 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 00:13 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 00:07 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 00:07 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 00:03 cwhite@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logging-hd2003.codfw.wmnet with reason: host reimage

2024-04-02

• 23:59 cwhite@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on logging-hd2003.codfw.wmnet with reason: host reimage
• 23:59 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 23:59 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 23:50 cwhite@cumin2002: START - Cookbook sre.hosts.reimage for host logging-hd2002.codfw.wmnet with OS bookworm
• 23:48 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 23:48 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 23:42 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 23:42 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 23:30 cwhite@cumin2002: START - Cookbook sre.hosts.reimage for host logging-hd2003.codfw.wmnet with OS bookworm
• 23:29 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 23:28 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 23:28 cwhite@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host logging-hd2003.codfw.wmnet with OS bookworm
• 23:22 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 23:22 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 23:15 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 23:15 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 23:09 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 23:09 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 23:03 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 23:03 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 22:58 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 22:57 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 22:52 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 22:52 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 22:46 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 22:46 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 22:42 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 22:42 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 22:40 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 22:40 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 22:38 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 22:38 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 22:35 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 22:35 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 22:33 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 22:33 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 22:31 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 22:31 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 22:31 cwhite@cumin2002: START - Cookbook sre.hosts.reimage for host logging-hd2003.codfw.wmnet with OS bookworm
• 22:29 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 22:29 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 22:28 cwhite@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host logging-hd2001.codfw.wmnet with OS bookworm
• 22:27 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 22:27 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 22:25 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 22:25 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 22:23 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 22:23 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 22:21 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 22:21 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 22:19 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 22:19 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 22:17 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 22:17 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 22:15 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 22:15 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 22:13 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 22:13 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 22:11 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 22:11 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 22:09 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 22:09 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 22:08 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 22:08 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 22:07 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 22:07 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 22:07 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 22:07 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 22:05 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 22:05 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 22:05 cwhite@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logging-hd2001.codfw.wmnet with reason: host reimage
• 22:03 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 22:03 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 22:02 cwhite@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on logging-hd2001.codfw.wmnet with reason: host reimage
• 22:01 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 22:01 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 21:58 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 21:58 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 21:56 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 21:56 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 21:54 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 21:54 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 21:52 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 21:52 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 21:50 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 21:50 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 21:50 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 21:49 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 21:49 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 21:48 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 21:48 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 21:46 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 21:46 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 21:44 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 21:44 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 21:42 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 21:42 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 21:40 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 21:40 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 21:38 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 21:38 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 21:38 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 21:38 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 21:36 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 21:36 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 21:36 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 21:34 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 21:34 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 21:34 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 21:32 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 21:32 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 21:32 cwhite@cumin2002: START - Cookbook sre.hosts.reimage for host logging-hd2001.codfw.wmnet with OS bookworm
• 21:31 cwhite@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host logging-hd2001.codfw.wmnet with OS bookworm
• 21:30 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 21:30 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 21:28 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 21:28 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 21:26 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 21:26 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 21:24 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 21:24 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 21:21 bking@cumin2002: END (PASS) - Cookbook sre.ganeti.resource-report (exit_code=0)
• 21:21 bking@cumin2002: START - Cookbook sre.ganeti.resource-report
• 21:21 bking@cumin2002: END (PASS) - Cookbook sre.ganeti.resource-report (exit_code=0)
• 21:21 bking@cumin2002: START - Cookbook sre.ganeti.resource-report
• 21:16 cwhite@cumin2002: START - Cookbook sre.hosts.reimage for host logging-hd2001.codfw.wmnet with OS bookworm
• 20:46 mutante: DNS - added new project language 'igl' - Igala is a Yoruboid language, spoken by the Igala ethnic group of Nigeria (800,000 speakers) T361644
• 20:13 sfaci@deploy1002: Finished deploy [airflow-dags/analytics@75163c7]: (no justification provided) (duration: 00m 16s)
• 20:13 sfaci@deploy1002: Started deploy [airflow-dags/analytics@75163c7]: (no justification provided)
• 20:07 sfaci@deploy1002: Finished deploy [airflow-dags/analytics@75163c7]: (no justification provided) (duration: 00m 46s)
• 20:06 sfaci@deploy1002: Started deploy [airflow-dags/analytics@75163c7]: (no justification provided)
• 20:06 sfaci@deploy1002: Finished deploy [airflow-dags/analytics@75163c7]: (no justification provided) (duration: 00m 54s)
• 20:05 sfaci@deploy1002: Started deploy [airflow-dags/analytics@75163c7]: (no justification provided)
• 18:48 cstone: civicrm upgraded from ed776060 to 5b7f9e06
• 17:26 cwhite@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host logging-hd2001.codfw.wmnet with OS bookworm
• 17:24 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
• 17:23 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
• 17:19 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1240.eqiad.wmnet with reason: Maintenance
• 17:19 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1240.eqiad.wmnet with reason: Maintenance
• 17:19 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1223 (T356166)', diff saved to https://phabricator.wikimedia.org/P59232 and previous config saved to /var/cache/conftool/dbconfig/20240402-171935-marostegui.json
• 17:13 Dreamy_Jazz: Creating cu_useragent table on WMF wikis - T359312
• 17:06 arnaudb@cumin1002: dbctl commit (dc=all): 'db2220 (re)pooling @ 100%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59230 and previous config saved to /var/cache/conftool/dbconfig/20240402-170625-arnaudb.json
• 17:06 arnaudb@cumin1002: dbctl commit (dc=all): 'db2219 (re)pooling @ 100%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59229 and previous config saved to /var/cache/conftool/dbconfig/20240402-170610-arnaudb.json
• 17:06 arnaudb@cumin1002: dbctl commit (dc=all): 'db2215 (re)pooling @ 100%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59228 and previous config saved to /var/cache/conftool/dbconfig/20240402-170603-arnaudb.json
• 17:05 arnaudb@cumin1002: dbctl commit (dc=all): 'db2214 (re)pooling @ 100%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59227 and previous config saved to /var/cache/conftool/dbconfig/20240402-170555-arnaudb.json
• 17:04 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P59226 and previous config saved to /var/cache/conftool/dbconfig/20240402-170427-marostegui.json
• 16:59 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 16:59 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 16:56 arnaudb@cumin1002: dbctl commit (dc=all): 'db1230 (re)pooling @ 100%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P59225 and previous config saved to /var/cache/conftool/dbconfig/20240402-165601-arnaudb.json
• 16:51 arnaudb@cumin1002: dbctl commit (dc=all): 'db2220 (re)pooling @ 75%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59224 and previous config saved to /var/cache/conftool/dbconfig/20240402-165119-arnaudb.json
• 16:51 arnaudb@cumin1002: dbctl commit (dc=all): 'db2219 (re)pooling @ 75%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59223 and previous config saved to /var/cache/conftool/dbconfig/20240402-165105-arnaudb.json
• 16:50 arnaudb@cumin1002: dbctl commit (dc=all): 'db2215 (re)pooling @ 75%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59222 and previous config saved to /var/cache/conftool/dbconfig/20240402-165058-arnaudb.json
• 16:50 arnaudb@cumin1002: dbctl commit (dc=all): 'db2214 (re)pooling @ 75%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59221 and previous config saved to /var/cache/conftool/dbconfig/20240402-165049-arnaudb.json
• 16:49 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P59220 and previous config saved to /var/cache/conftool/dbconfig/20240402-164920-marostegui.json
• 16:45 jgiannelos@deploy1002: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
• 16:44 jgiannelos@deploy1002: helmfile [staging] START helmfile.d/services/mobileapps: apply
• 16:40 arnaudb@cumin1002: dbctl commit (dc=all): 'db1230 (re)pooling @ 75%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P59219 and previous config saved to /var/cache/conftool/dbconfig/20240402-164055-arnaudb.json
• 16:40 jgiannelos@deploy1002: helmfile [staging] START helmfile.d/services/mobileapps: apply
• 16:38 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 16:38 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 16:36 arnaudb@cumin1002: dbctl commit (dc=all): 'db2220 (re)pooling @ 50%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59218 and previous config saved to /var/cache/conftool/dbconfig/20240402-163613-arnaudb.json
• 16:36 arnaudb@cumin1002: dbctl commit (dc=all): 'db2219 (re)pooling @ 50%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59217 and previous config saved to /var/cache/conftool/dbconfig/20240402-163559-arnaudb.json
• 16:35 arnaudb@cumin1002: dbctl commit (dc=all): 'db2215 (re)pooling @ 50%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59216 and previous config saved to /var/cache/conftool/dbconfig/20240402-163552-arnaudb.json
• 16:35 arnaudb@cumin1002: dbctl commit (dc=all): 'db2214 (re)pooling @ 50%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59215 and previous config saved to /var/cache/conftool/dbconfig/20240402-163544-arnaudb.json
• 16:34 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1223 (T356166)', diff saved to https://phabricator.wikimedia.org/P59214 and previous config saved to /var/cache/conftool/dbconfig/20240402-163413-marostegui.json
• 16:28 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 16:27 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 16:25 arnaudb@cumin1002: dbctl commit (dc=all): 'db1230 (re)pooling @ 50%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P59213 and previous config saved to /var/cache/conftool/dbconfig/20240402-162550-arnaudb.json
• 16:22 aborrero@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1036.eqiad.wmnet with OS bookworm
• 16:21 arnaudb@cumin1002: dbctl commit (dc=all): 'db2220 (re)pooling @ 25%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59212 and previous config saved to /var/cache/conftool/dbconfig/20240402-162107-arnaudb.json
• 16:20 arnaudb@cumin1002: dbctl commit (dc=all): 'db2219 (re)pooling @ 25%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59211 and previous config saved to /var/cache/conftool/dbconfig/20240402-162053-arnaudb.json
• 16:20 arnaudb@cumin1002: dbctl commit (dc=all): 'db2215 (re)pooling @ 25%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59210 and previous config saved to /var/cache/conftool/dbconfig/20240402-162046-arnaudb.json
• 16:20 arnaudb@cumin1002: dbctl commit (dc=all): 'db2214 (re)pooling @ 25%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59209 and previous config saved to /var/cache/conftool/dbconfig/20240402-162038-arnaudb.json
• 16:13 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov1006.eqiad.wmnet with reason: host reimage
• 16:11 vriley@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov1006.eqiad.wmnet with reason: host reimage
• 16:10 arnaudb@cumin1002: dbctl commit (dc=all): 'db1230 (re)pooling @ 25%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P59208 and previous config saved to /var/cache/conftool/dbconfig/20240402-161044-arnaudb.json
• 16:06 arnaudb@cumin1002: dbctl commit (dc=all): 'db2220 (re)pooling @ 16%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59207 and previous config saved to /var/cache/conftool/dbconfig/20240402-160602-arnaudb.json
• 16:05 arnaudb@cumin1002: dbctl commit (dc=all): 'db2219 (re)pooling @ 16%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59206 and previous config saved to /var/cache/conftool/dbconfig/20240402-160547-arnaudb.json
• 16:05 arnaudb@cumin1002: dbctl commit (dc=all): 'db2215 (re)pooling @ 16%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59205 and previous config saved to /var/cache/conftool/dbconfig/20240402-160540-arnaudb.json
• 16:05 arnaudb@cumin1002: dbctl commit (dc=all): 'db2214 (re)pooling @ 16%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59204 and previous config saved to /var/cache/conftool/dbconfig/20240402-160532-arnaudb.json
• 16:04 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 16:04 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 16:02 jnuche@deploy1002: Finished scap: Backport for rest: add default null to nullable typed prop (T361577) (duration: 14m 39s)
• 16:02 cwhite@cumin2002: START - Cookbook sre.hosts.reimage for host logging-hd2001.codfw.wmnet with OS bookworm
• 15:57 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 15:57 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 15:57 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 15:57 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host dbprov1006.eqiad.wmnet with OS bullseye
• 15:55 aborrero@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1036
• 15:55 arnaudb@cumin1002: dbctl commit (dc=all): 'db1230 (re)pooling @ 15%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P59203 and previous config saved to /var/cache/conftool/dbconfig/20240402-155538-arnaudb.json
• 15:55 aborrero@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1036
• 15:54 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 15:54 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 15:54 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 15:51 aborrero@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1036.eqiad.wmnet with reason: host reimage
• 15:50 arnaudb@cumin1002: dbctl commit (dc=all): 'db2220 (re)pooling @ 8%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59202 and previous config saved to /var/cache/conftool/dbconfig/20240402-155056-arnaudb.json
• 15:50 arnaudb@cumin1002: dbctl commit (dc=all): 'db2219 (re)pooling @ 8%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59201 and previous config saved to /var/cache/conftool/dbconfig/20240402-155042-arnaudb.json
• 15:50 arnaudb@cumin1002: dbctl commit (dc=all): 'db2215 (re)pooling @ 8%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59200 and previous config saved to /var/cache/conftool/dbconfig/20240402-155035-arnaudb.json
• 15:50 arnaudb@cumin1002: dbctl commit (dc=all): 'db2214 (re)pooling @ 8%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59199 and previous config saved to /var/cache/conftool/dbconfig/20240402-155026-arnaudb.json
• 15:50 jnuche@deploy1002: samtar and jnuche: Continuing with sync
• 15:50 jnuche@deploy1002: samtar and jnuche: Backport for rest: add default null to nullable typed prop (T361577) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 15:49 aborrero@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1036.eqiad.wmnet with reason: host reimage
• 15:47 jnuche@deploy1002: Started scap: Backport for rest: add default null to nullable typed prop (T361577)
• 15:40 arnaudb@cumin1002: dbctl commit (dc=all): 'db1230 (re)pooling @ 10%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P59198 and previous config saved to /var/cache/conftool/dbconfig/20240402-154033-arnaudb.json
• 15:35 arnaudb@cumin1002: dbctl commit (dc=all): 'db2220 (re)pooling @ 4%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59197 and previous config saved to /var/cache/conftool/dbconfig/20240402-153550-arnaudb.json
• 15:35 arnaudb@cumin1002: dbctl commit (dc=all): 'db2219 (re)pooling @ 4%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59196 and previous config saved to /var/cache/conftool/dbconfig/20240402-153536-arnaudb.json
• 15:35 arnaudb@cumin1002: dbctl commit (dc=all): 'db2215 (re)pooling @ 4%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59195 and previous config saved to /var/cache/conftool/dbconfig/20240402-153529-arnaudb.json
• 15:35 arnaudb@cumin1002: dbctl commit (dc=all): 'db2214 (re)pooling @ 4%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59194 and previous config saved to /var/cache/conftool/dbconfig/20240402-153521-arnaudb.json
• 15:31 aborrero@cumin1002: START - Cookbook sre.hosts.reimage for host cloudvirt1036.eqiad.wmnet with OS bookworm
• 15:25 arnaudb@cumin1002: dbctl commit (dc=all): 'db1230 (re)pooling @ 5%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P59193 and previous config saved to /var/cache/conftool/dbconfig/20240402-152527-arnaudb.json
• 15:23 jgiannelos@deploy1002: Finished deploy [restbase/deploy@c4d19d7]: (no justification provided) (duration: 00m 16s)
• 15:23 jgiannelos@deploy1002: Started deploy [restbase/deploy@c4d19d7]: (no justification provided)
• 15:21 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1230.eqiad.wmnet with OS bookworm
• 15:20 arnaudb@cumin1002: dbctl commit (dc=all): 'db2220 (re)pooling @ 2%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59192 and previous config saved to /var/cache/conftool/dbconfig/20240402-152044-arnaudb.json
• 15:20 arnaudb@cumin1002: dbctl commit (dc=all): 'db2219 (re)pooling @ 2%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59191 and previous config saved to /var/cache/conftool/dbconfig/20240402-152031-arnaudb.json
• 15:20 arnaudb@cumin1002: dbctl commit (dc=all): 'db2215 (re)pooling @ 2%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59190 and previous config saved to /var/cache/conftool/dbconfig/20240402-152023-arnaudb.json
• 15:20 arnaudb@cumin1002: dbctl commit (dc=all): 'db2214 (re)pooling @ 2%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59189 and previous config saved to /var/cache/conftool/dbconfig/20240402-152015-arnaudb.json
• 15:12 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1223 (T356166)', diff saved to https://phabricator.wikimedia.org/P59188 and previous config saved to /var/cache/conftool/dbconfig/20240402-151235-marostegui.json
• 15:12 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1223.eqiad.wmnet with reason: Maintenance
• 15:12 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1223.eqiad.wmnet with reason: Maintenance
• 15:12 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1212 (T356166)', diff saved to https://phabricator.wikimedia.org/P59187 and previous config saved to /var/cache/conftool/dbconfig/20240402-151213-marostegui.json
• 15:05 arnaudb@cumin1002: dbctl commit (dc=all): 'db2220 (re)pooling @ 1%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59186 and previous config saved to /var/cache/conftool/dbconfig/20240402-150538-arnaudb.json
• 15:05 arnaudb@cumin1002: dbctl commit (dc=all): 'db2219 (re)pooling @ 1%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59185 and previous config saved to /var/cache/conftool/dbconfig/20240402-150525-arnaudb.json
• 15:05 arnaudb@cumin1002: dbctl commit (dc=all): 'db2215 (re)pooling @ 1%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59184 and previous config saved to /var/cache/conftool/dbconfig/20240402-150516-arnaudb.json
• 15:05 arnaudb@cumin1002: dbctl commit (dc=all): 'db2214 (re)pooling @ 1%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59183 and previous config saved to /var/cache/conftool/dbconfig/20240402-150509-arnaudb.json
• 15:01 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1230.eqiad.wmnet with reason: host reimage
• 14:57 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1230.eqiad.wmnet with reason: host reimage
• 14:57 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P59182 and previous config saved to /var/cache/conftool/dbconfig/20240402-145705-marostegui.json
• 14:56 moritzm: installing mariadb security updates (as packaged in Debian, unrelated to the wmf-mariadb packages)
• 14:45 arnaudb@cumin1002: START - Cookbook sre.hosts.reimage for host db1230.eqiad.wmnet with OS bookworm
• 14:42 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1230.eqiad.wmnet with reason: Silence for reimaging
• 14:42 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1230.eqiad.wmnet with reason: Silence for reimaging
• 14:42 arnaudb@cumin1002: dbctl commit (dc=all): 'Depool to reimage db1230', diff saved to https://phabricator.wikimedia.org/P59181 and previous config saved to /var/cache/conftool/dbconfig/20240402-144221-arnaudb.json
• 14:41 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P59180 and previous config saved to /var/cache/conftool/dbconfig/20240402-144158-marostegui.json
• 14:38 fabfur@cumin1002: conftool action : set/pooled=yes; selector: name=cp3066.esams.wmnet
• 14:38 fabfur: repooling cp3066 after reimage (T360430)
• 14:29 arnaudb@cumin1002: dbctl commit (dc=all): 'db1185 (re)pooling @ 100%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P59179 and previous config saved to /var/cache/conftool/dbconfig/20240402-142859-arnaudb.json
• 14:28 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3066.esams.wmnet with OS bullseye
• 14:26 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1212 (T356166)', diff saved to https://phabricator.wikimedia.org/P59178 and previous config saved to /var/cache/conftool/dbconfig/20240402-142650-marostegui.json
• 14:15 marostegui@cumin1002: dbctl commit (dc=all): 'db1188 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P59177 and previous config saved to /var/cache/conftool/dbconfig/20240402-141541-root.json
• 14:13 arnaudb@cumin1002: dbctl commit (dc=all): 'db1185 (re)pooling @ 75%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P59176 and previous config saved to /var/cache/conftool/dbconfig/20240402-141353-arnaudb.json
• 14:09 moritzm: imported jenkins 2.440.2 to thirdparty/ci for buster-wikimedia T360759
• 14:05 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3066.esams.wmnet with reason: host reimage
• 14:02 fabfur@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3066.esams.wmnet with reason: host reimage
• 14:00 marostegui@cumin1002: dbctl commit (dc=all): 'db1188 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P59174 and previous config saved to /var/cache/conftool/dbconfig/20240402-140035-root.json
• 13:58 arnaudb@cumin1002: dbctl commit (dc=all): 'db1185 (re)pooling @ 50%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P59173 and previous config saved to /var/cache/conftool/dbconfig/20240402-135847-arnaudb.json
• 13:45 marostegui@cumin1002: dbctl commit (dc=all): 'db1188 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P59172 and previous config saved to /var/cache/conftool/dbconfig/20240402-134528-root.json
• 13:43 arnaudb@cumin1002: dbctl commit (dc=all): 'db1185 (re)pooling @ 25%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P59171 and previous config saved to /var/cache/conftool/dbconfig/20240402-134342-arnaudb.json
• 13:38 TheresNoTime: closing UTC afternoon backport window
• 13:38 fabfur@cumin1002: START - Cookbook sre.hosts.reimage for host cp3066.esams.wmnet with OS bullseye
• 13:37 samtar@deploy1002: Finished scap: Backport for InitialiseSettings: Enable Edit Recovery on all projects (T355548) (duration: 16m 26s)
• 13:33 fabfur@cumin1002: conftool action : set/pooled=no; selector: name=cp3066.esams.wmnet
• 13:32 fabfur: depool cp3066 for reimage (T360430)
• 13:30 marostegui@cumin1002: dbctl commit (dc=all): 'db1188 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P59170 and previous config saved to /var/cache/conftool/dbconfig/20240402-133023-root.json
• 13:28 arnaudb@cumin1002: dbctl commit (dc=all): 'db1185 (re)pooling @ 15%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P59169 and previous config saved to /var/cache/conftool/dbconfig/20240402-132836-arnaudb.json
• 13:25 samtar@deploy1002: samtar: Continuing with sync
• 13:23 samtar@deploy1002: samtar: Backport for InitialiseSettings: Enable Edit Recovery on all projects (T355548) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 13:21 samtar@deploy1002: Started scap: Backport for InitialiseSettings: Enable Edit Recovery on all projects (T355548)
• 13:18 dreamyjazz@deploy1002: Finished scap: Backport for Deploy partial action blocks everywhere (T353496) (duration: 15m 33s)
• 13:15 marostegui@cumin1002: dbctl commit (dc=all): 'db1188 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P59168 and previous config saved to /var/cache/conftool/dbconfig/20240402-131517-root.json
• 13:13 arnaudb@cumin1002: dbctl commit (dc=all): 'db1185 (re)pooling @ 10%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P59167 and previous config saved to /var/cache/conftool/dbconfig/20240402-131330-arnaudb.json
• 13:05 dreamyjazz@deploy1002: dreamyjazz and tchanders: Continuing with sync
• 13:05 dreamyjazz@deploy1002: dreamyjazz and tchanders: Backport for Deploy partial action blocks everywhere (T353496) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 13:04 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1212 (T356166)', diff saved to https://phabricator.wikimedia.org/P59166 and previous config saved to /var/cache/conftool/dbconfig/20240402-130423-marostegui.json
• 13:04 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
• 13:04 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 16:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
• 13:03 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1212.eqiad.wmnet with reason: Maintenance
• 13:03 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1212.eqiad.wmnet with reason: Maintenance
• 13:03 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T356166)', diff saved to https://phabricator.wikimedia.org/P59165 and previous config saved to /var/cache/conftool/dbconfig/20240402-130341-marostegui.json
• 13:02 dreamyjazz@deploy1002: Started scap: Backport for Deploy partial action blocks everywhere (T353496)
• 13:00 marostegui@cumin1002: dbctl commit (dc=all): 'db1188 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P59164 and previous config saved to /var/cache/conftool/dbconfig/20240402-130012-root.json
• 12:58 arnaudb@cumin1002: dbctl commit (dc=all): 'db1185 (re)pooling @ 5%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P59163 and previous config saved to /var/cache/conftool/dbconfig/20240402-125825-arnaudb.json
• 12:48 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P59162 and previous config saved to /var/cache/conftool/dbconfig/20240402-124834-marostegui.json
• 12:45 marostegui@cumin1002: dbctl commit (dc=all): 'db1188 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P59161 and previous config saved to /var/cache/conftool/dbconfig/20240402-124506-root.json
• 12:44 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1188.eqiad.wmnet with OS bookworm
• 12:39 aborrero@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1035.eqiad.wmnet with OS bookworm
• 12:33 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P59160 and previous config saved to /var/cache/conftool/dbconfig/20240402-123326-marostegui.json
• 12:28 taavi: taavi@deploy1002 ~ $ sudo systemctl kill train-presync.service # T361580
• 12:22 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1188.eqiad.wmnet with reason: host reimage
• 12:20 hnowlan@deploy2002: helmfile [eqiad] [main] DONE helmfile.d/services/mw-jobrunner : sync
• 12:19 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1188.eqiad.wmnet with reason: host reimage
• 12:19 hnowlan@deploy2002: helmfile [eqiad] [canary] DONE helmfile.d/services/mw-jobrunner : sync
• 12:18 hnowlan@deploy2002: helmfile [eqiad] [main] START helmfile.d/services/mw-jobrunner : sync
• 12:18 hnowlan@deploy2002: helmfile [eqiad] [canary] START helmfile.d/services/mw-jobrunner : sync
• 12:18 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T356166)', diff saved to https://phabricator.wikimedia.org/P59159 and previous config saved to /var/cache/conftool/dbconfig/20240402-121819-marostegui.json
• 12:13 moritzm: installing pillow security updates
• 12:13 hnowlan@deploy2002: helmfile [codfw] [main] DONE helmfile.d/services/mw-jobrunner : sync
• 12:12 hnowlan@deploy2002: helmfile [codfw] [canary] DONE helmfile.d/services/mw-jobrunner : sync
• 12:11 hnowlan@deploy2002: helmfile [codfw] [main] START helmfile.d/services/mw-jobrunner : sync
• 12:11 hnowlan@deploy2002: helmfile [codfw] [canary] START helmfile.d/services/mw-jobrunner : sync
• 12:09 aborrero@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1035.eqiad.wmnet with reason: host reimage
• 12:07 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db1188.eqiad.wmnet with OS bookworm
• 12:06 aborrero@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1035.eqiad.wmnet with reason: host reimage
• 12:04 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1188 T361543', diff saved to https://phabricator.wikimedia.org/P59158 and previous config saved to /var/cache/conftool/dbconfig/20240402-120455-root.json
• 11:58 aborrero@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1035
• 11:58 aborrero@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1035
• 11:49 aborrero@cumin1002: START - Cookbook sre.hosts.reimage for host cloudvirt1035.eqiad.wmnet with OS bookworm
• 11:33 sfaci@deploy1002: helmfile [eqiad] DONE helmfile.d/services/editor-analytics: apply
• 11:32 sfaci@deploy1002: helmfile [eqiad] START helmfile.d/services/editor-analytics: apply
• 11:32 sfaci@deploy1002: helmfile [codfw] DONE helmfile.d/services/editor-analytics: apply
• 11:32 sfaci@deploy1002: helmfile [codfw] START helmfile.d/services/editor-analytics: apply
• 11:31 sfaci@deploy1002: helmfile [staging] DONE helmfile.d/services/editor-analytics: apply
• 11:31 sfaci@deploy1002: helmfile [staging] START helmfile.d/services/editor-analytics: apply
• 11:29 sfaci@deploy1002: helmfile [eqiad] DONE helmfile.d/services/edit-analytics: apply
• 11:29 sfaci@deploy1002: helmfile [eqiad] START helmfile.d/services/edit-analytics: apply
• 11:29 sfaci@deploy1002: helmfile [codfw] DONE helmfile.d/services/edit-analytics: apply
• 11:28 sfaci@deploy1002: helmfile [codfw] START helmfile.d/services/edit-analytics: apply
• 11:25 sfaci@deploy1002: helmfile [staging] DONE helmfile.d/services/edit-analytics: apply
• 11:25 sfaci@deploy1002: helmfile [staging] START helmfile.d/services/edit-analytics: apply
• 11:01 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1198 (T356166)', diff saved to https://phabricator.wikimedia.org/P59157 and previous config saved to /var/cache/conftool/dbconfig/20240402-110122-marostegui.json
• 11:01 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1198.eqiad.wmnet with reason: Maintenance
• 11:01 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1198.eqiad.wmnet with reason: Maintenance
• 11:01 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T356166)', diff saved to https://phabricator.wikimedia.org/P59156 and previous config saved to /var/cache/conftool/dbconfig/20240402-110100-marostegui.json
• 10:45 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P59155 and previous config saved to /var/cache/conftool/dbconfig/20240402-104552-marostegui.json
• 10:30 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P59154 and previous config saved to /var/cache/conftool/dbconfig/20240402-103045-marostegui.json
• 10:29 marostegui@cumin1002: dbctl commit (dc=all): 'db1197 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P59153 and previous config saved to /var/cache/conftool/dbconfig/20240402-102951-root.json
• 10:15 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T356166)', diff saved to https://phabricator.wikimedia.org/P59152 and previous config saved to /var/cache/conftool/dbconfig/20240402-101538-marostegui.json
• 10:14 marostegui@cumin1002: dbctl commit (dc=all): 'db1197 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P59151 and previous config saved to /var/cache/conftool/dbconfig/20240402-101445-root.json
• 09:59 marostegui@cumin1002: dbctl commit (dc=all): 'db1197 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P59150 and previous config saved to /var/cache/conftool/dbconfig/20240402-095939-root.json
• 09:53 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1185.eqiad.wmnet with OS bookworm
• 09:46 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) Will create a clone of db2115.codfw.wmnet onto db2215.codfw.wmnet
• 09:44 marostegui@cumin1002: dbctl commit (dc=all): 'db1197 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P59149 and previous config saved to /var/cache/conftool/dbconfig/20240402-094433-root.json
• 09:32 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1185.eqiad.wmnet with reason: host reimage
• 09:29 marostegui@cumin1002: dbctl commit (dc=all): 'db1197 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P59148 and previous config saved to /var/cache/conftool/dbconfig/20240402-092928-root.json
• 09:28 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1185.eqiad.wmnet with reason: host reimage
• 09:22 jnuche@deploy1002: rebuilt and synchronized wikiversions files: group0 wikis to 1.42.0-wmf.25 refs T360157
• 09:15 arnaudb@cumin1002: START - Cookbook sre.hosts.reimage for host db1185.eqiad.wmnet with OS bookworm
• 09:14 marostegui@cumin1002: dbctl commit (dc=all): 'db1197 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P59147 and previous config saved to /var/cache/conftool/dbconfig/20240402-091422-root.json
• 09:14 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1185.eqiad.wmnet with reason: Silence for reimaging
• 09:13 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1185.eqiad.wmnet with reason: Silence for reimaging
• 09:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Depool to reimage db1185', diff saved to https://phabricator.wikimedia.org/P59146 and previous config saved to /var/cache/conftool/dbconfig/20240402-091303-arnaudb.json
• 09:02 jnuche@deploy1002: Finished scap: testwikis wikis to 1.42.0-wmf.25 refs T360157 (duration: 51m 03s)
• 08:59 marostegui@cumin1002: dbctl commit (dc=all): 'db1197 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P59145 and previous config saved to /var/cache/conftool/dbconfig/20240402-085917-root.json
• 08:58 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1189 (T356166)', diff saved to https://phabricator.wikimedia.org/P59144 and previous config saved to /var/cache/conftool/dbconfig/20240402-085814-marostegui.json
• 08:58 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1189.eqiad.wmnet with reason: Maintenance
• 08:57 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1189.eqiad.wmnet with reason: Maintenance
• 08:57 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T356166)', diff saved to https://phabricator.wikimedia.org/P59143 and previous config saved to /var/cache/conftool/dbconfig/20240402-085752-marostegui.json
• 08:53 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1197.eqiad.wmnet with OS bookworm
• 08:42 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P59142 and previous config saved to /var/cache/conftool/dbconfig/20240402-084244-marostegui.json
• 08:32 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1197.eqiad.wmnet with reason: host reimage
• 08:28 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1197.eqiad.wmnet with reason: host reimage
• 08:27 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P59141 and previous config saved to /var/cache/conftool/dbconfig/20240402-082737-marostegui.json
• 08:06 marostegui@cumin1002: dbctl commit (dc=all): 'db1229 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P59137 and previous config saved to /var/cache/conftool/dbconfig/20240402-080642-root.json
• 08:02 godog: restore SRE business hours routing/escalation after the holidays - T350192
• 07:51 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 38278
• 07:51 marostegui@cumin1002: dbctl commit (dc=all): 'db1229 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P59134 and previous config saved to /var/cache/conftool/dbconfig/20240402-075136-root.json
• 07:49 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 38278
• 07:47 hashar: UTC morning backport window completed
• 07:46 hashar@deploy1002: Finished scap: Backport for webrequest: disable canary events. (T314956 T351117) (duration: 34m 03s)
• 07:39 taavi: update firewall policy on cr-eqiad, cr-codfw T361537
• 07:36 marostegui@cumin1002: dbctl commit (dc=all): 'db1229 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P59133 and previous config saved to /var/cache/conftool/dbconfig/20240402-073631-root.json
• 07:30 hashar@deploy1002: gmodena and hashar: Continuing with sync
• 07:28 hashar@deploy1002: gmodena and hashar: Backport for webrequest: disable canary events. (T314956 T351117) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 07:21 marostegui@cumin1002: dbctl commit (dc=all): 'db1229 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P59132 and previous config saved to /var/cache/conftool/dbconfig/20240402-072125-root.json
• 07:12 hashar@deploy1002: Started scap: Backport for webrequest: disable canary events. (T314956 T351117)
• 07:06 marostegui@cumin1002: dbctl commit (dc=all): 'db1229 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P59131 and previous config saved to /var/cache/conftool/dbconfig/20240402-070619-root.json
• 07:03 moritzm: installing util-linux security updates
• 06:57 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1175 (T356166)', diff saved to https://phabricator.wikimedia.org/P59130 and previous config saved to /var/cache/conftool/dbconfig/20240402-065751-marostegui.json
• 06:57 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1175.eqiad.wmnet with reason: Maintenance
• 06:57 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1175.eqiad.wmnet with reason: Maintenance
• 06:57 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T356166)', diff saved to https://phabricator.wikimedia.org/P59129 and previous config saved to /var/cache/conftool/dbconfig/20240402-065728-marostegui.json
• 06:51 marostegui@cumin1002: dbctl commit (dc=all): 'db1229 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P59128 and previous config saved to /var/cache/conftool/dbconfig/20240402-065113-root.json
• 06:42 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P59127 and previous config saved to /var/cache/conftool/dbconfig/20240402-064221-marostegui.json
• 06:36 marostegui@cumin1002: dbctl commit (dc=all): 'db1229 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P59126 and previous config saved to /var/cache/conftool/dbconfig/20240402-063607-root.json
• 06:35 marostegui@cumin1002: dbctl commit (dc=all): 'es1024 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P59125 and previous config saved to /var/cache/conftool/dbconfig/20240402-063510-root.json
• 06:27 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P59124 and previous config saved to /var/cache/conftool/dbconfig/20240402-062713-marostegui.json
• 06:24 marostegui@cumin1002: dbctl commit (dc=all): 'db2115 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P59123 and previous config saved to /var/cache/conftool/dbconfig/20240402-062427-root.json
• 06:23 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1229.eqiad.wmnet with OS bookworm
• 06:20 marostegui@cumin1002: dbctl commit (dc=all): 'es1024 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P59122 and previous config saved to /var/cache/conftool/dbconfig/20240402-062004-root.json
• 06:12 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T356166)', diff saved to https://phabricator.wikimedia.org/P59121 and previous config saved to /var/cache/conftool/dbconfig/20240402-061206-marostegui.json
• 06:09 marostegui@cumin1002: dbctl commit (dc=all): 'db2115 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P59120 and previous config saved to /var/cache/conftool/dbconfig/20240402-060922-root.json
• 06:04 marostegui@cumin1002: dbctl commit (dc=all): 'es1024 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P59119 and previous config saved to /var/cache/conftool/dbconfig/20240402-060459-root.json
• 06:03 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1229.eqiad.wmnet with reason: host reimage
• 05:59 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1229.eqiad.wmnet with reason: host reimage
• 05:54 marostegui@cumin1002: dbctl commit (dc=all): 'db2115 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P59118 and previous config saved to /var/cache/conftool/dbconfig/20240402-055416-root.json
• 05:49 marostegui@cumin1002: dbctl commit (dc=all): 'es1024 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P59117 and previous config saved to /var/cache/conftool/dbconfig/20240402-054953-root.json
• 05:46 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db1229.eqiad.wmnet with OS bookworm
• 05:44 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1229 T361543', diff saved to https://phabricator.wikimedia.org/P59116 and previous config saved to /var/cache/conftool/dbconfig/20240402-054408-root.json
• 05:39 marostegui@cumin1002: dbctl commit (dc=all): 'db2115 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P59115 and previous config saved to /var/cache/conftool/dbconfig/20240402-053910-root.json
• 05:34 marostegui@cumin1002: dbctl commit (dc=all): 'es1024 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P59114 and previous config saved to /var/cache/conftool/dbconfig/20240402-053447-root.json
• 05:24 marostegui@cumin1002: dbctl commit (dc=all): 'db2115 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P59113 and previous config saved to /var/cache/conftool/dbconfig/20240402-052404-root.json
• 05:19 marostegui@cumin1002: dbctl commit (dc=all): 'es1024 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P59112 and previous config saved to /var/cache/conftool/dbconfig/20240402-051942-root.json
• 05:09 marostegui@cumin1002: dbctl commit (dc=all): 'db2115 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P59111 and previous config saved to /var/cache/conftool/dbconfig/20240402-050859-root.json
• 05:04 marostegui@cumin1002: dbctl commit (dc=all): 'es1024 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P59110 and previous config saved to /var/cache/conftool/dbconfig/20240402-050436-root.json
• 04:57 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1166 (T356166)', diff saved to https://phabricator.wikimedia.org/P59109 and previous config saved to /var/cache/conftool/dbconfig/20240402-045716-marostegui.json
• 04:57 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1166.eqiad.wmnet with reason: Maintenance
• 04:56 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1166.eqiad.wmnet with reason: Maintenance
• 04:56 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es1024 T358746', diff saved to https://phabricator.wikimedia.org/P59108 and previous config saved to /var/cache/conftool/dbconfig/20240402-045559-root.json
• 04:53 marostegui@cumin1002: dbctl commit (dc=all): 'db2115 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P59107 and previous config saved to /var/cache/conftool/dbconfig/20240402-045353-root.json
• 04:53 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1150.eqiad.wmnet with reason: Maintenance
• 04:53 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1150.eqiad.wmnet with reason: Maintenance
• 03:03 mwpresync@deploy1002: Pruned MediaWiki: 1.42.0-wmf.22 (duration: 03m 20s)

2024-04-01

• 23:14 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dbprov1006.eqiad.wmnet with OS bullseye
• 22:41 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
• 22:41 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
• 22:31 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1245.eqiad.wmnet with reason: Maintenance
• 22:31 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1245.eqiad.wmnet with reason: Maintenance
• 22:30 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1230 (T356166)', diff saved to https://phabricator.wikimedia.org/P59105 and previous config saved to /var/cache/conftool/dbconfig/20240401-223055-marostegui.json
• 22:15 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1230', diff saved to https://phabricator.wikimedia.org/P59104 and previous config saved to /var/cache/conftool/dbconfig/20240401-221548-marostegui.json
• 22:00 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1230', diff saved to https://phabricator.wikimedia.org/P59103 and previous config saved to /var/cache/conftool/dbconfig/20240401-220040-marostegui.json
• 21:53 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host dbprov1006.eqiad.wmnet with OS bullseye
• 21:45 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1230 (T356166)', diff saved to https://phabricator.wikimedia.org/P59102 and previous config saved to /var/cache/conftool/dbconfig/20240401-214532-marostegui.json
• 21:38 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1230 (T356166)', diff saved to https://phabricator.wikimedia.org/P59101 and previous config saved to /var/cache/conftool/dbconfig/20240401-213834-marostegui.json
• 21:38 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1230.eqiad.wmnet with reason: Maintenance
• 21:38 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1230.eqiad.wmnet with reason: Maintenance
• 21:28 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1216.eqiad.wmnet with reason: Maintenance
• 21:27 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1216.eqiad.wmnet with reason: Maintenance
• 21:27 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1213 (T356166)', diff saved to https://phabricator.wikimedia.org/P59100 and previous config saved to /var/cache/conftool/dbconfig/20240401-212751-marostegui.json
• 21:12 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1213', diff saved to https://phabricator.wikimedia.org/P59099 and previous config saved to /var/cache/conftool/dbconfig/20240401-211244-marostegui.json
• 21:03 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dbprov1006.eqiad.wmnet with OS bullseye
• 20:57 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1213', diff saved to https://phabricator.wikimedia.org/P59098 and previous config saved to /var/cache/conftool/dbconfig/20240401-205736-marostegui.json
• 20:42 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1213 (T356166)', diff saved to https://phabricator.wikimedia.org/P59097 and previous config saved to /var/cache/conftool/dbconfig/20240401-204229-marostegui.json
• 20:36 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic2088.codfw.wmnet with OS bullseye
• 20:32 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1213 (T356166)', diff saved to https://phabricator.wikimedia.org/P59096 and previous config saved to /var/cache/conftool/dbconfig/20240401-203254-marostegui.json
• 20:32 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1213.eqiad.wmnet with reason: Maintenance
• 20:32 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1213.eqiad.wmnet with reason: Maintenance
• 20:32 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1210 (T356166)', diff saved to https://phabricator.wikimedia.org/P59095 and previous config saved to /var/cache/conftool/dbconfig/20240401-203232-marostegui.json
• 20:19 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic2088.codfw.wmnet with reason: host reimage
• 20:17 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1210', diff saved to https://phabricator.wikimedia.org/P59094 and previous config saved to /var/cache/conftool/dbconfig/20240401-201725-marostegui.json
• 20:17 bking@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on elastic2088.codfw.wmnet with reason: host reimage
• 20:02 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1210', diff saved to https://phabricator.wikimedia.org/P59093 and previous config saved to /var/cache/conftool/dbconfig/20240401-200217-marostegui.json
• 20:00 bking@cumin2002: START - Cookbook sre.hosts.reimage for host elastic2088.codfw.wmnet with OS bullseye
• 19:47 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1210 (T356166)', diff saved to https://phabricator.wikimedia.org/P59092 and previous config saved to /var/cache/conftool/dbconfig/20240401-194709-marostegui.json
• 19:42 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host dbprov1006.eqiad.wmnet with OS bullseye
• 19:37 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1210 (T356166)', diff saved to https://phabricator.wikimedia.org/P59091 and previous config saved to /var/cache/conftool/dbconfig/20240401-193713-marostegui.json
• 19:37 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1210.eqiad.wmnet with reason: Maintenance
• 19:36 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1210.eqiad.wmnet with reason: Maintenance
• 19:36 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1200 (T356166)', diff saved to https://phabricator.wikimedia.org/P59090 and previous config saved to /var/cache/conftool/dbconfig/20240401-193650-marostegui.json
• 19:21 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P59089 and previous config saved to /var/cache/conftool/dbconfig/20240401-192143-marostegui.json
• 19:06 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P59088 and previous config saved to /var/cache/conftool/dbconfig/20240401-190635-marostegui.json
• 18:51 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1200 (T356166)', diff saved to https://phabricator.wikimedia.org/P59087 and previous config saved to /var/cache/conftool/dbconfig/20240401-185128-marostegui.json
• 18:44 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1200 (T356166)', diff saved to https://phabricator.wikimedia.org/P59086 and previous config saved to /var/cache/conftool/dbconfig/20240401-184455-marostegui.json
• 18:44 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1200.eqiad.wmnet with reason: Maintenance
• 18:44 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1200.eqiad.wmnet with reason: Maintenance
• 18:44 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1185 (T356166)', diff saved to https://phabricator.wikimedia.org/P59085 and previous config saved to /var/cache/conftool/dbconfig/20240401-184432-marostegui.json
• 18:29 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P59084 and previous config saved to /var/cache/conftool/dbconfig/20240401-182924-marostegui.json
• 18:14 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P59083 and previous config saved to /var/cache/conftool/dbconfig/20240401-181417-marostegui.json
• 17:59 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1185 (T356166)', diff saved to https://phabricator.wikimedia.org/P59082 and previous config saved to /var/cache/conftool/dbconfig/20240401-175910-marostegui.json
• 17:58 mutante: LDAP - removed uid migr from groups nda and wmde (T361266)
• 17:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1185 (T356166)', diff saved to https://phabricator.wikimedia.org/P59081 and previous config saved to /var/cache/conftool/dbconfig/20240401-175300-marostegui.json
• 17:52 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1185.eqiad.wmnet with reason: Maintenance
• 17:52 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1185.eqiad.wmnet with reason: Maintenance
• 17:52 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T356166)', diff saved to https://phabricator.wikimedia.org/P59080 and previous config saved to /var/cache/conftool/dbconfig/20240401-175237-marostegui.json
• 17:37 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P59079 and previous config saved to /var/cache/conftool/dbconfig/20240401-173729-marostegui.json
• 17:22 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P59078 and previous config saved to /var/cache/conftool/dbconfig/20240401-172221-marostegui.json
• 17:07 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T356166)', diff saved to https://phabricator.wikimedia.org/P59077 and previous config saved to /var/cache/conftool/dbconfig/20240401-170713-marostegui.json
• 16:56 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1161 (T356166)', diff saved to https://phabricator.wikimedia.org/P59076 and previous config saved to /var/cache/conftool/dbconfig/20240401-165559-marostegui.json
• 16:55 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
• 16:55 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 16:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
• 16:55 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1161.eqiad.wmnet with reason: Maintenance
• 16:55 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1161.eqiad.wmnet with reason: Maintenance
• 16:43 eevans@deploy1002: Finished deploy [cassandra/logstash-logback-encoder@42653e6]: (no justification provided) (duration: 00m 33s)
• 16:42 eevans@deploy1002: Started deploy [cassandra/logstash-logback-encoder@42653e6]: (no justification provided)
• 16:07 Dreamy_Jazz: Restarting MediaModeration scanning script - https://wikitech.wikimedia.org/wiki/MediaModeration
• 16:05 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.dhcp (exit_code=99) for host elastic2088.codfw.wmnet
• 15:50 pt1979@cumin2002: START - Cookbook sre.hosts.dhcp for host elastic2088.codfw.wmnet
• 15:36 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov1005.eqiad.wmnet with OS bullseye
• 15:36 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
• 15:35 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
• 15:33 urandom: cassandra (restbase): re-enable blocking read-repair — T354561
• 15:30 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp4052.ulsfo.wmnet,service=(cdn|ats-be)
• 15:14 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov1005.eqiad.wmnet with reason: host reimage
• 15:13 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4052.ulsfo.wmnet with OS bullseye
• 15:11 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
• 15:11 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov1005.eqiad.wmnet with reason: host reimage
• 15:11 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
• 15:11 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1231 (T356166)', diff saved to https://phabricator.wikimedia.org/P59075 and previous config saved to /var/cache/conftool/dbconfig/20240401-151114-marostegui.json
• 14:58 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host dbprov1005.eqiad.wmnet with OS bullseye
• 14:56 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P59074 and previous config saved to /var/cache/conftool/dbconfig/20240401-145606-marostegui.json
• 14:52 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage
• 14:52 jhancock@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['elastic2088']
• 14:51 jhancock@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['elastic2088']
• 14:49 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage
• 14:41 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P59073 and previous config saved to /var/cache/conftool/dbconfig/20240401-144059-marostegui.json
• 14:32 jhancock@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['elastic2088']
• 14:32 jhancock@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['elastic2088']
• 14:28 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dbprov1005.eqiad.wmnet with OS bullseye
• 14:27 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp4052.ulsfo.wmnet with OS bullseye
• 14:26 sukhe@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4052.ulsfo.wmnet with OS bullseye
• 14:25 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1231 (T356166)', diff saved to https://phabricator.wikimedia.org/P59072 and previous config saved to /var/cache/conftool/dbconfig/20240401-142552-marostegui.json
• 14:06 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp4052.ulsfo.wmnet with OS bullseye
• 14:06 sukhe: reimage cp4052 back to bullseye
• 13:52 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1231 (T356166)', diff saved to https://phabricator.wikimedia.org/P59071 and previous config saved to /var/cache/conftool/dbconfig/20240401-135204-marostegui.json
• 13:51 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1231.eqiad.wmnet with reason: Maintenance
• 13:51 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1231.eqiad.wmnet with reason: Maintenance
• 13:11 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host dbprov1005.eqiad.wmnet with OS bullseye
• 12:55 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1225.eqiad.wmnet with reason: Maintenance
• 12:55 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1225.eqiad.wmnet with reason: Maintenance
• 12:55 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1224 (T356166)', diff saved to https://phabricator.wikimedia.org/P59070 and previous config saved to /var/cache/conftool/dbconfig/20240401-125524-marostegui.json
• 12:40 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1224', diff saved to https://phabricator.wikimedia.org/P59069 and previous config saved to /var/cache/conftool/dbconfig/20240401-124017-marostegui.json
• 12:25 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1224', diff saved to https://phabricator.wikimedia.org/P59068 and previous config saved to /var/cache/conftool/dbconfig/20240401-122510-marostegui.json
• 12:10 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1224 (T356166)', diff saved to https://phabricator.wikimedia.org/P59067 and previous config saved to /var/cache/conftool/dbconfig/20240401-121002-marostegui.json
• 11:41 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1224 (T356166)', diff saved to https://phabricator.wikimedia.org/P59066 and previous config saved to /var/cache/conftool/dbconfig/20240401-114105-marostegui.json
• 11:40 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1224.eqiad.wmnet with reason: Maintenance
• 11:40 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1224.eqiad.wmnet with reason: Maintenance
• 11:40 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T356166)', diff saved to https://phabricator.wikimedia.org/P59065 and previous config saved to /var/cache/conftool/dbconfig/20240401-114043-marostegui.json
• 11:25 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P59064 and previous config saved to /var/cache/conftool/dbconfig/20240401-112536-marostegui.json
• 11:10 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P59063 and previous config saved to /var/cache/conftool/dbconfig/20240401-111028-marostegui.json
• 10:55 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T356166)', diff saved to https://phabricator.wikimedia.org/P59062 and previous config saved to /var/cache/conftool/dbconfig/20240401-105521-marostegui.json
• 10:23 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1201 (T356166)', diff saved to https://phabricator.wikimedia.org/P59061 and previous config saved to /var/cache/conftool/dbconfig/20240401-102328-marostegui.json
• 10:23 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1201.eqiad.wmnet with reason: Maintenance
• 10:23 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1201.eqiad.wmnet with reason: Maintenance
• 10:23 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T356166)', diff saved to https://phabricator.wikimedia.org/P59060 and previous config saved to /var/cache/conftool/dbconfig/20240401-102306-marostegui.json
• 10:07 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P59059 and previous config saved to /var/cache/conftool/dbconfig/20240401-100758-marostegui.json
• 09:52 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P59058 and previous config saved to /var/cache/conftool/dbconfig/20240401-095251-marostegui.json
• 09:37 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T356166)', diff saved to https://phabricator.wikimedia.org/P59057 and previous config saved to /var/cache/conftool/dbconfig/20240401-093744-marostegui.json
• 09:05 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1187 (T356166)', diff saved to https://phabricator.wikimedia.org/P59056 and previous config saved to /var/cache/conftool/dbconfig/20240401-090527-marostegui.json
• 09:05 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1187.eqiad.wmnet with reason: Maintenance
• 09:05 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1187.eqiad.wmnet with reason: Maintenance
• 09:05 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T356166)', diff saved to https://phabricator.wikimedia.org/P59055 and previous config saved to /var/cache/conftool/dbconfig/20240401-090503-marostegui.json
• 08:49 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P59054 and previous config saved to /var/cache/conftool/dbconfig/20240401-084956-marostegui.json
• 08:34 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P59053 and previous config saved to /var/cache/conftool/dbconfig/20240401-083449-marostegui.json
• 07:42 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1180 (T356166)', diff saved to https://phabricator.wikimedia.org/P59051 and previous config saved to /var/cache/conftool/dbconfig/20240401-074221-marostegui.json
• 07:42 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1180.eqiad.wmnet with reason: Maintenance
• 07:42 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1180.eqiad.wmnet with reason: Maintenance
• 07:42 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T356166)', diff saved to https://phabricator.wikimedia.org/P59050 and previous config saved to /var/cache/conftool/dbconfig/20240401-074158-marostegui.json
• 07:26 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P59049 and previous config saved to /var/cache/conftool/dbconfig/20240401-072650-marostegui.json
• 07:11 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P59048 and previous config saved to /var/cache/conftool/dbconfig/20240401-071143-marostegui.json
• 06:56 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T356166)', diff saved to https://phabricator.wikimedia.org/P59047 and previous config saved to /var/cache/conftool/dbconfig/20240401-065635-marostegui.json
• 06:29 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1168 (T356166)', diff saved to https://phabricator.wikimedia.org/P59046 and previous config saved to /var/cache/conftool/dbconfig/20240401-062954-marostegui.json
• 06:29 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1168.eqiad.wmnet with reason: Maintenance
• 06:29 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1168.eqiad.wmnet with reason: Maintenance
• 06:29 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T356166)', diff saved to https://phabricator.wikimedia.org/P59045 and previous config saved to /var/cache/conftool/dbconfig/20240401-062932-marostegui.json
• 06:14 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P59044 and previous config saved to /var/cache/conftool/dbconfig/20240401-061423-marostegui.json
• 05:59 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P59043 and previous config saved to /var/cache/conftool/dbconfig/20240401-055915-marostegui.json
• 05:44 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T356166)', diff saved to https://phabricator.wikimedia.org/P59042 and previous config saved to /var/cache/conftool/dbconfig/20240401-054408-marostegui.json
• 05:14 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1165 (T356166)', diff saved to https://phabricator.wikimedia.org/P59041 and previous config saved to /var/cache/conftool/dbconfig/20240401-051402-marostegui.json
• 05:13 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
• 05:13 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 16:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
• 05:13 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1165.eqiad.wmnet with reason: Maintenance
• 05:13 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1165.eqiad.wmnet with reason: Maintenance
• 05:03 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on 9 hosts with reason: Fixing intermediate master
• 05:03 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 5:00:00 on 9 hosts with reason: Fixing intermediate master

Archives

See Server Admin Log/Archives.