minimal implementation of care partner alerts #715
Conversation
ewollesen
force-pushed
the
eric-cpa-alerts
branch
from
374bae2
to
2deda23
Compare
ewollesen
force-pushed
the
eric-cpa-alerts
branch
2 times, most recently
from
8549c33
to
8367902
Compare
ewollesen
force-pushed
the
eric-cpa-alerts
branch
from
8367902
to
2ea9686
Compare
ewollesen
changed the title
ewollesen
force-pushed
the
eric-cpa-alerts
branch
from
2ea9686
to
7246848
Compare
|
To use this in QA, it must be paired with tidepool-org/hydrophone#145 and tidepool-org/go-common#64 |
ewollesen
force-pushed
the
eric-cpa-alerts
branch
2 times, most recently
from
c50d589
to
986106b
Compare
There was a problem hiding this comment.
Looks good overall, but the retry mechanism which is implemented here doesn't satisfy the latency requirements. The current implementation is ok for internal usage, but it's not production ready. This could be handled in a separate PR if this makes the development and QA process easier.
data/events/events.go
Outdated
| } | ||
| handler := asyncevents.NewSaramaConsumerGroupHandler(&asyncevents.NTimesRetryingConsumer{ | ||
| Consumer: r.Config.MessageConsumer, | ||
| Delay: CappedExponentialBinaryDelay(AlertsEventRetryDelayMaximum), |
There was a problem hiding this comment.
I don't think this is a suitable retry strategy given the latency requirements for this service. Kafka's consumer group concurrency is limited to the number of partitions of the topic. This number cannot be very high because Kafka's memory consumption grows linearly with the number of partitions. From this follows that the number of partitions is much lower than the number of users we will have and the data of multiple users will end up in the same partition. A failure to evaluate a single user's alerts for one minute as currently set by the CappedExponentialBinaryDelay will introduce at least a minute delay to all of the users sharing the same partition, because messages in a single partition are processed serially.
Alert notifications should be near real-time - up to 10 seconds latency is acceptable. I think the solution proposed in this design document is how this should be handled. Other solutions which satisfy the requirements are welcome.
There was a problem hiding this comment.
This will require some more in-depth thought on my part... Will do.
There was a problem hiding this comment.
Yeah, I think you're right, let's get this review merged, and I'll work on getting a multiple topic solution set up. Given the flexibility we have now, it shouldn't be too bad.
There was a problem hiding this comment.
I have the multi-tier retry in the eric-alerts-multi-topic-retry branch.
ewollesen
force-pushed
the
eric-cpa-alerts
branch
2 times, most recently
from
c08e1fc
to
967c617
Compare
ewollesen
force-pushed
the
eric-cpa-alerts
branch
from
967c617
to
cd42b19
Compare
ewollesen
force-pushed
the
eric-cpa-alerts
branch
2 times, most recently
from
9432468
to
eaa652e
Compare
ewollesen
force-pushed
the
eric-cpa-alerts
branch
from
d6449a1
to
ee5da4a
Compare
|
@toddkazakov I just removed two config env vars. I believe we talked about that before, but it slipped my mind until I was reviewing the helm chart changes today, where they came up again. So the re-review here is just around the config parsing, in the most recent commit of the PR, nothing else is changed. |
The Get endpoint already exists on the service, so only the List endpoint needed to be added there. BACK-2554
Through discussions it was confirmed that Repeat is not universal to all alerts. So it's lifted out of the Base alert and re-inserted into those alerts where it should be present (namely Low and High alerts only). BACK-2554
These activity properties will track the times at which alerts were sent, resolved, or acknowledged. BACK-2554
This endpoint will be used by upcoming changes to the auth client to allow care partner backend processes to retrieve device tokens in order to send mobile device push notifications. BACK-2554
This functionality will be used by care partner processes to retrieve device tokens in order to send mobile device push notifications in response to care partner alerts being triggered. BACK-2554
This was missed when moving device tokens from the data service to the auth service in commit a0f5a84. BACK-2554
Basic steps are taken to allow for other push notification services to be easily added in the future. BACK-2554
So that sarama log messages better follow our standards, and will be emitted as JSON when log.Logger is configured for that. Before this change, the sarama logs were printed in plain-text without any of the benefits of the platform log.Logger. BACK-2554
The existing FaultTolerantConsumer isn't used because it's retry semantics are hard-wired and aren't compatible with what care partner alerting's needs. Note: A proper implementation of AlertsEventsConsumer to consume events is yet to be written. It will follow shortly. BACK-2554
The upload id is necessary to ensure that only the proper device data uploads are evaluated for care partner alert conditions. BACK-2554
If the necessary configuration isn't found, then push notifications will instead be logged. BACK-2554
These methods return Note objects that can be sent as push notifications. NotLooping evaluation will be handled in a later commit. BACK-2554
It uses the new asyncevents from go-common, as alerts processing requires different retry semantics than the existing solution. The Pusher interface is moved out of data/service into data/events to avoid a circular dependency. BACK-2554
No longer needed
In response to request during code review.
As caught by Todd in code review. BACK-2554
When a care partner alert encounters an error, the message is moved to a separate topic that will cause it to be retried after a delay. Any number of these topics can be configured. BACK-2499
Instead of a static delay, uses a "not before" time found in a Kafka message header. Consumption of the message will not be attempted until the time has passed. This allows for more accurate delays, as the time required to process an earlier message doesn't further delay the current message's processing. BACK-2449
These won't be changing at runtime, so there's no need to complicate the initialization by making these configurable. The topic's prefix is configurable, and that's the part that will change from environment to environment at runtime. BACK-2554
ewollesen
force-pushed
the
eric-cpa-alerts
branch
from
8ae1dc8
to
28fdf06
Compare
I've attempted to organize the commits to ease reviewing.
This used to be a series of PRs, but that didn't really work out. They're all collapsed into this one.
Shouldn't be merged until tidepool-org/go-common#64 is merged, then this should have it's go-common bumped.
BACK-2554