Mongo connection not recovered

[lawrencegripper]

Mgo looks like it doesn't automatically recover from a dropped mongo connection, this affects the front API.

~/go/src/github.com/lawrencegripper/ion [lg/fix157|✚ 1⚑ 1] 
14:57 $ curl --header "Content-Type: application/json"   --request POST   --data '{"url": "http://download.blender.org/peach/bigbuckbunny_movies/BigBuckBunny_320x180.mp4"}'   http://localhost:9001/
Failed writing to document store
✔ ~/go/src/github.com/lawrencegripper/ion [lg/fix157|✚ 1⚑ 1] 
14:57 $ k get pods 
NAME                                 READY     STATUS    RESTARTS   AGE
downloader-bdgr6-797f9fb69-64kmd     1/1       Running   0          38m
ion-front-api-5d656b444f-gmtzh       1/1       Running   0          39m
ion-management-api-d9b55655d-4tdqk   1/1       Running   0          39m
transcode-bdgr6-75d4dfcfc8-rs729     1/1       Running   0          38m
✔ ~/go/src/github.com/lawrencegripper/ion [lg/fix157|✚ 1⚑ 1] 
14:57 $ k logs ion-front-api-5d656b444f-gmtzh 
time="2018-08-01T13:18:45Z" level=warning msg="Can't read management config from file %s ../../configs/frontapi.yaml" error="open ../../configs/frontapi.yaml: no such file or directory"
time="2018-08-01T13:18:45Z" level=info msg="Initialising AMQP connection"
time="2018-08-01T13:18:49Z" level=info msg="Initialising Mongo connection"
time="2018-08-01T13:18:49Z" level=info msg="Starting api server"
time="2018-08-01T13:18:49Z" level=info msg="Starting listening: 0.0.0.0:9001"
time="2018-08-01T13:20:09Z" level=info msg="Processing URL: http://download.blender.org/peach/bigbuckbunny_movies/BigBuckBunny_320x180.mp4"
time="2018-08-01T13:20:09Z" level=info msg="Publishing event /frontapi.new_link"
time="2018-08-01T13:20:09Z" level=info msg="Event published"
time="2018-08-01T13:57:03Z" level=info msg="Processing URL: http://download.blender.org/peach/bigbuckbunny_movies/BigBuckBunny_320x180.mp4"
time="2018-08-01T13:57:03Z" level=error msg="failed to add context '{Context:0xc4202d82d0 ParentEventID: Files:[] Data:[{Key:url Value:http://download.blender.org/peach/bigbuckbunny_movies/BigBuckBunny_320x180.mp4}]}' with error 'error creates document: EOF'"
time="2018-08-01T13:57:04Z" level=info msg="Processing URL: http://download.blender.org/peach/bigbuckbunny_movies/BigBuckBunny_320x180.mp4"
time="2018-08-01T13:57:04Z" level=error msg="failed to add context '{Context:0xc4202d88c0 ParentEventID: Files:[] Data:[{Key:url Value:http://download.blender.org/peach/bigbuckbunny_movies/BigBuckBunny_320x180.mp4}]}' with error 'error creates document: EOF'"
time="2018-08-01T13:57:05Z" level=info msg="Processing URL: http://download.blender.org/peach/bigbuckbunny_movies/BigBuckBunny_320x180.mp4"
time="2018-08-01T13:57:05Z" level=error msg="failed to add context '{Context:0xc42041e1e0 ParentEventID: Files:[] Data:[{Key:url Value:http://download.blender.org/peach/bigbuckbunny_movies/BigBuckBunny_320x180.mp4}]}' with error 'error creates document: EOF'"
time="2018-08-01T13:57:06Z" level=info msg="Processing URL: http://download.blender.org/peach/bigbuckbunny_movies/BigBuckBunny_320x180.mp4"
time="2018-08-01T13:57:06Z" level=error msg="failed to add context '{Context:0xc4202d8f00 ParentEventID: Files:[] Data:[{Key:url Value:http://download.blender.org/peach/bigbuckbunny_movies/BigBuckBunny_320x180.mp4}]}' with error 'error creates document: EOF'"
time="2018-08-01T13:57:41Z" level=info msg="Processing URL: http://download.blender.org/peach/bigbuckbunny_movies/BigBuckBunny_320x180.mp4"
time="2018-08-01T13:57:41Z" level=error msg="failed to add context '{Context:0xc42041efa0 ParentEventID: Files:[] Data:[{Key:url Value:http://download.blender.org/peach/bigbuckbunny_movies/BigBuckBunny_320x180.mp4}]}' with error 'error creates document: EOF'"

[lawrencegripper]

This also affects the dispatcher when writing logs for the items

time="2018-08-01T14:04:11Z" level=error msg="failed to store logs for job in metastore" endTime="2018-08-01T14:03:59.316864Z" error="error creates document: EOF" message="failed to store logs for job in metastore" messageID=59179d5fba354a21b68d3ea71212571b startTime="2018-08-01T14:03:20.16158Z" taskDurationSec=0x107ed30 taskID=0xc4201ed890 taskName=0xc4201ed8a0 taskNodeID=0xc4201edbe0 taskPoolID=0xc4201edbd0
time="2018-08-01T14:04:11Z" level=error msg="failed to log to logstore" endTime="2018-08-01T14:03:59.316864Z" error="error creates document: EOF" message="failed to log to logstore" messageID=59179d5fba354a21b68d3ea71212571b startTime="2018-08-01T14:03:20.16158Z" taskDurationSec=0x107ed30 taskID=0xc4201ed890 taskName=0xc4201ed8a0 taskNodeID=0xc4201edbe0 taskPoolID=0xc4201edbd0

[lawrencegripper]

One solution would be to panic either component when this occurs, another would be to investigate if mgo_v2 support some kind of automated recovery from a lost connection