Background: Who is the Client and what is the product/service?
Context: Scaling “too fast”
At guild.xyz the team is full of talented developers.
The traction and the iterations are very fast, and the product decisions are shaped mostly by the investors and bigger “clients” feedbacks and needs. This is totally OK for an early-stage startup, but guild.xyz grew soo fast, that the need for proper user metrics emerged soon.
(part of) The Solution: Setting up a product funnel
My research usually pointed in one direction: the main challenge in the guild.xyz-s onboarding is to teach users new concepts (eg.: roles and requirements on guild page) and their relations to existing entities (eg.: already existing Discord roles).
Guild.xyz has two main User Persona:
- Community owners (2% of the traffic)
- They will set up and manage the Guild page
- They need to understand the whole process
- They will communicate about it to the community members
- They want to make community management easier but in the same time they fear losing members who don’t understand / like the new access management flow
- Potential / existing community members (98% of the traffic)
- They see the link for the Guild page and enter to the community after a verification process
- If they don't trust the platform or don't understand something, they churn off or ask for help usually from the Community owner.
I wanted to focus more on the latter. They are our “real-users”, and mediators to the members. This is why the understanding “Guild Creation Flow” was in higher priority than the “Join Guild Flow”, however, we gained lots of actionable insight, (than conversion improvement) from the latter also.
Steps:
1. Research and explore the flow.
Having real experience with the service is important, even if you think, you understand the whole process. Usually, I start every project with a product/service walkthrough, while I record everything and create screenshots, writing down raw insights. During my criminal psychology studies, once a judge was invited as a lecturer. He said, that before the trial of a murder case, he often goes for a walk to the crime scene, because the first hand experience always adds insights. The same is true for digital products.
2. Draft out the measurement points and discuss with stakeholders
After assessing needs with stakeholder interviews I made a draft funnel. The main goal was to create a medium for conversation. Thinking together is always easier visually.
Step 3. Consult with the devs about custom event possibilities.
The tool we used, was more like an infrastructure-measurement system, instead of product analytics. So I had to adapt to this and facilitate the solution for creating custom events.
Step 4. Review and test the events, whether they really work as intended.
Of course, it doesn't. :) It's tricky, because sometimes the metrics look like they working, but some of the numbers make no sense. But with the collaboration of a dev collage, we were able to find and fix measurement bugs and anomalies.
Step 5. Build a workflow for curating custom events.
Sounds obvious, but it isn’t. This product changes fast. If we have an event that is outdated, we can't just delete it, because it is possible that we want to use it in the future for comparisons.
A possible solution for this is the very accurate tracking of the custom events and their timelines. In our case the medium for this was a gist table, curated by the dev who creates new events:
Step 6. Discover some new insights.
After setting a metric system up, and waiting for the first significant sample size ( shortly after you weed out every bug) is full of excitement. Similar to waiting for the first conversion rates after a campaign you designed, but more intense.
We found lots of things, but the main insight I want to show here were:
- Our join flow’s weak points are the third-party services. Both Community owners and Community members need to connect their Ethereum wallet and their Discord. Weeding out the bugs about Wallet connect and developing an in-site Discord join button seemed a low-hanging fruit.
Step 7. Write an in-house Notion guide about the queries.
My goal was more than deliver useful insights. I wanted to introduce data-driven product decisions to the whole organization. This is why I made this internal documentation.
Step 8. Results
Shortly after the launch of bug fixes and features ( eg.: onsite Discord Join) and the new Onboarding flow, the success ratio was 66% higher on the Guild Creation Flow.
After waiting for a bigger sample size the improvement….improved to 18-20%.
The Join Guild Flow liked the changes also. Before:
After: