GitLab v1 integration: What to measure?

[nickoferrall]

I'd like to add analytics to the GitLab integration to learn how users use it and what we can improve.

Specifically, I'd like to know whether they're happy with the v1 search filter or whether we should build the following more advanced search filter:

In v1, the search input filters the GitLab issues based on whether the query string matches the issue title. See gif below:

I imagine the advanced filtering would be helpful to users, but I don't know that for sure. I do know that it will be hard to implement! See v2 milestone with issues relating to this: https://github.com/ParabolInc/parabol/milestone/126

So, I'd love to find out whether users want this before building it. To do that, I think we could:

1. Speak with users once v1 has shipped
2. Analyse the data

What shall we track to learn whether users want fancy filtering?

• Track the start of the search. Record the user id, timestamp, and meeting id. We could do this when they first enter a character rather than when they focus on the input as they could click on the input by accident.
• Track the end of the search. Record the user id, timestamp, and meeting id. This could be when the user leaves the input. With start & end timestamps, we could calculate the median time spent searching. If it's a few seconds, that's probably good. If it's 10+ seconds, that's probably not great.
• Track when a user selects an issue in the Scope phase. Record the current search query string, number of projects selected in the project filter, user id, timestamp, and meeting id. This will give us an idea of how often users search for issues before selecting them.

I don't expect there to be one metric that tells us whether to build it or not, but the data should help us make a decision.

What else should we track?

• When a new issue is created. Record the user id, timestamp, and meeting id. This could be when the user submits the new issue input.
• When a project filter menu item is selected. Record the user id, timestamp, and meeting id.
• When the select all button is used. Record the user id, timestamp, and meeting id. If it's not being used, we could remove it.

I'd love to know whether @tghanken and @avivapinchas feel that these are the right things to track and whether there's anything else we should track? Once you're both happy with the plan, I'll create an issue and add it to the GitLab v1 milestone.

[tghanken]

I think this is a good start. If the goal is to identify tension within the search panel, I think there's a few things that make sense to add:

• An event each time a "search" occurs. If the user opens the panel, and searches four or five different times before selecting cards, that's a good indicator that our current solution isn't working for them.
• An event each time the input box is cleared within a session, as another proxy for user frustration.

When we start looking at trace style events like this, passing a trace_id of some sort to the event would also make it much easier for us to analyze downstream. As an example, we could generate a guid when the GitLab integration interface is originally pulled up, then add that to all following events until the integration is closed. This way, if they reopen the integration screen a second time for the same meeting, we can track the individual integration "sessions" separately.

[nickoferrall]

Sounds good! I've created an issue: #6303

Thanks for your input!

[nickoferrall]

Hey @tghanken, now that I'm implementing this issue, I'm slightly unclear on the following:

we could generate a guid when the GitLab integration interface is originally pulled up, then add that to all following events until the integration is closed. This way, if they reopen the integration screen a second time for the same meeting, we can track the individual integration "sessions" separately

The GitLab integration auth token doesn't expire so it's unlikely that the user will reopen the integration screen. We are tracking the userId and the meetingId, so we'll be able to track the user's behaviour in each meeting. Do you think that will get us what we need?

[tghanken]

I guess "Integration Screen" was probably the wrong way to describe it. What I really meant was every time they selected the "GitLab" integration tab while in the "Scope" phase. If we drop the guid, we'd lose some of our ability to gauge frustration on long running poker meetings, where they might revisit and add cards to be scoped over multiple days.

Overall, I don't think this is a dealbreaker, and userId and meetingId gets us ~90% of what we would want to look at.

[nickoferrall]

OK cool, I'll press on with userId and meetingId for now to get v1 shipped, and we can add trace_id down the line if we think it'd be helpful.

[tghanken]

Sounds good!

[avivapinchas]

I think this makes sense!

If we track who has the integration configured, we can use that as a list of folks to reach out to for further user research.