Troubleshoot—Caitsith service crash after upgrade to 5.2.1

{
  "severity":"EMERGENCY",
  "ts":1648146980.2922893,
  "msg":"Unable to split collections",
  "srv":"caitsith",
  "error":"unable to shard collection yeul.cloudpublicipaddress: A previous attempt to shard collection yeul cloudpublicipaddress failed after writing some initial chunks to config.chunks. Please manually delete the partially written chunks for collection yeul.cloudpublicipaddress from config.chunks"
}

Activate the voila environment, issue command mgos

In the Mongo shell, list and remove the chunks for yeul service

mongos> use config
switched to db config
mongos> db.chunks.find({ns: "yeul.cloudpublicipaddress"})
{ "_id" : ObjectId("623a4da7fe3ca025ef7ba098"), "ns" : "yeul.cloudpublicipaddress", "min" : { "zone" : { "$minKey" : 1 }, "zhash" : { "$minKey" : 1 } }, "max" : { "zone" : 0, "zhash" : { "$minKey" : 1 } }, "shard" : "reports-z1-0", "lastmod" : Timestamp(1, 0), "lastmodEpoch" : ObjectId("623a4d9f59173339ee3477e4"), "history" : [ { "validAfter" : Timestamp(1647988127, 56), "shard" : "reports-z1-0" } ] }
{ "_id" : ObjectId("623a4da7fe3ca025ef7ba097"), "ns" : "yeul.cloudpublicipaddress", "min" : { "zone" : 0, "zhash" : { "$minKey" : 1 } }, "max" : { "zone" : 1, "zhash" : { "$maxKey" : 1 } }, "shard" : "shard-z0-0", "lastmod" : Timestamp(1, 1), "lastmodEpoch" : ObjectId("623a4d9f59173339ee3477e4"), "history" : [ { "validAfter" : Timestamp(1647988127, 56), "shard" : "shard-z0-0" } ] }
{ "_id" : ObjectId("623a4da7fe3ca025ef7ba096"), "ns" : "yeul.cloudpublicipaddress", "min" : { "zone" : 1, "zhash" : { "$maxKey" : 1 } }, "max" : { "zone" : { "$maxKey" : 1 }, "zhash" : { "$maxKey" : 1 } }, "shard" : "shard-z0-0", "lastmod" : Timestamp(1, 2), "lastmodEpoch" : ObjectId("623a4d9f59173339ee3477e4"), "history" : [ { "validAfter" : Timestamp(1647988127, 56), "shard" : "shard-z0-0" } ] }
mongos> db.chunks.deleteMany({ns: "yeul.cloudpublicipaddress"})
{ "acknowledged" : true, "deletedCount" : 3 }
mongos> exit
Bye

In the voila environment, restart the service using restart caitsith

voila - (release-5.2.1) (backend-5.2.1)
 ➤ restart caitsith
The following resources will be restarted in a rolling fashion:
 - deployment/caitsith
Type any key to confirm (control-c to cancel):
deployment.apps/caitsith restarted

Confirm that caitsith has resumed operations

voila - (release-5.2.1) (backend-5.2.1)
 ➤ k get pods -l app=caitsith
NAME                        READY   STATUS    RESTARTS   AGE
caitsith-57c77857b4-p7dfc   1/1     Running   0          25s
voila - (release-5.2.1) (backend-rc-31410)
 ➤ k logs caitsith-57c77857b4-p7dfc
◼︎ CAITSITH v1.2050.3 (56968f4) release-5.2.1
{"severity":"INFO","ts":1648251958.057413,"msg":"GOMAXPROCS configured","srv":"caitsith","max":32}
{"severity":"INFO","ts":1648251958.061174,"msg":"Discovering platform","srv":"caitsith","api":"https://wutai-internal"}
{"severity":"INFO","ts":1648251958.0711584,"msg":"License verified","srv":"caitsith","company":"Palo Alto Networks, Inc","enforcers":-1,"pus":-1,"exp":"16 Jun 24 04:09 UTC"}
{"severity":"INFO","ts":1648251958.072206,"msg":"OpenTracing enabled","srv":"caitsith","server":"jaeger-agent:6831"}
{"severity":"INFO","ts":1648251958.072241,"msg":"Health server started","srv":"caitsith","address":":1080"}
{"severity":"INFO","ts":1648251958.072244,"msg":"Sharding management","srv":"caitsith","enabled":true}
{"severity":"INFO","ts":1648251958.0725868,"msg":"Attribute encryption","srv":"caitsith","status":"enabled"}
{"severity":"INFO","ts":1648251958.0796413,"msg":"Connected to mongodb","srv":"caitsith","url":"mongodb://mongodb-shard-router-0.mongodb-shard-router,mongodb-shard-router-1.mongodb-shard-router,mongodb-shard-router-2.mongodb-shard-router?authMechanism=MONGODB-X509","db":"caitsith"}
{"severity":"INFO","ts":1648251958.080971,"msg":"Database versionning aligned","srv":"caitsith"}
{"severity":"INFO","ts":1648251958.0949416,"msg":"Attribute encryption","srv":"caitsith","status":"enabled"}
voila - (release-5.2.1) (backend-5.2.1)
 ➤