Back to work

BeforeChat-only alpha

AfterTalent profiles

54%Limited betafaster to a match

Limited beta

LyRise · AI Talent Matcher · 2023

From chat replies
to confident choices.

LyRise's first AI matcher answered in plain text, and employers could not tell which candidate to pick. I redesigned it around talent profiles. In the limited beta, time to a match was 54% lower than on the chat-only alpha.

Role
Product designer, full-time, reporting to the CEO
Company
LyRise, AI startup
Year
2023
Tools
Figma

Objective

Improve the conversion from talent request initiation to successful talent interview booking.

  • 54%decrease in time to match
  • 19%higher confidence in selection
  • ~300users in the limited beta in its first month

01 · Context

Rebuilding the platform around LLMs

LyRise is a platform for matching employers in the US and EU with AI talent in Africa and South-east Asia. ChatGPT and other LLMs had just launched, so we were rebuilding our platform with the integration of LLMs.

In an agile startup, we adopted a mentality of always putting users first, and putting our product in front of users as soon as possible. So I created an MVP feature list that I extrapolated from our core function's user flow, and our dev team developed an alpha version with minimal UI, a prototype almost.

The alpha · a chat with minimal UI
The chat-only alpha: the assistant's reply is a paragraph per candidate, with a summary and the file paths it was read from. Names are blurred.

MVP feature list

Must have

  • LyRiseAI LLM chat interface
  • Talent database
  • Talent shortlisting
  • Interview booking

Delighters

  • Chat to job request auto fill
  • Full talent profiles (viewed by asking the LLM for more info about a talent)

02 · Problem discovery

Testing the alpha with users

After the alpha version was developed, I started testing the product with users. I conducted:

28 usability tests

I conducted 28 remote usability tests. The objective of the usability test was to have a user initiate a chat with the LLM to find an AI talent for a pre-defined role. I wanted to see:

  • how users interacted with the LLM
  • how long it took them to find an AI talent they would want to interview
  • what steps they would like to see next
A remote usability test on a video call: the call's title, the tester's shared screen with the chat-only alpha, the in-call chat with the task they were given, the tester's camera tile, blurred, and the interviewer's. The candidate names in the alpha's reply and the test links are blurred.

28 surveys

For those same users from the usability tests, I would fill out a post-usability test survey to document some qualitative and quantitative data. Some of those were:

  • whether the user found an AI talent they want to interview or not
  • if they did, how confident were they in their selection?
  • if they didn't, why didn't they?

LyRiseAI Alpha post-usability test survey

This survey is filled by the UX researcher for each usability test, the form itself is updated as per usability test notes.

Was the user able to successfully find an AI talent they were interested in interviewing?

YesNoOther...

If yes, how long did it take?

Short answer text

If yes, how confident is the user in their selection?

1 Very unconfident2 Somewhat unconfident3 Neutral4 Somewhat confident5 Very confident
Other...

If no, why?

Too little information to determineAll recommended talents were not qualifiedCould not prompt the LLM properlyOther...

Did the LLM hallucinate at any point?

YesNoOther...

If yes, after how many prompts did the LLM begin hallucinating?

Short answer text

16 user interviews

A few months earlier, I had conducted 16 user interviews with our main target persona to understand their process for hiring and what criteria they assess when looking for an AI talent for an AI role.

A user interview on a video call: the call's title, the participant's camera tile, blurred, and the interviewer's.

User interview questions bank

User interview answers

Could you briefly go over your hiring process?

We typically start with a detailed job description outlining the specific AI skills and experience needed for the role. We then post the job on various platforms and review incoming resumes. Shortlisted candidates go through a technical screening, followed by multiple rounds of interviews with the head of HR, me (CTO) and sometimes the CEO as well.

What is the most painful stage in that process?

Sourcing qualified AI talent for sure. There's a huge gap between the number of AI roles and qualified candidates, so it's incredibly time-consuming to find candidates with the right mix of skills and experience.

What criteria do you have when reviewing applicant profiles?

I usually look for relevant experience in AI, specific skills for the required role. I also review their GitHub repositories, this is the most important factor for me. Cultural fit, problem solving abilities and potential for growth are also important factors.

A CTO, in one of the interviews

The feedback from these activities also served in finding the solutions.

03 · The problems

Three key problems

Usability tests and user interviews showed us we had three key problems:

01

Time consuming

Users were taking too long to figure out how to properly prompt the LLM to recommend AI talent. By my estimates, it would take the user roughly 3 to 4 minutes just trying to figure out how to prompt the LLM.

02

Low confidence

Users didn't feel too confident in the recommended AI talent because they didn't have enough information until they asked the LLM for more info.

03

Low recall

When users asked a chain of follow-up questions to an original prompt, the model would begin hallucinating by the third or fourth follow-up at best.

04 · The solutions

Three key features

To address these issues we implemented three key features:

01

Conversation design

The goal of conversation design is to decrease the unpredictability of user prompts by having the LLM guide the user by asking specific questions, and once the LLM has sufficient info, it will make a recommendation.

02

Full talent profile

The most common reason for low confidence in users' selection was lack of access to the talent's full CV and experience. So I, together with the dev team, worked on creating a full talent profile that includes all the criteria I gathered from user interviews with our target persona a few months earlier. Talents had to stay anonymous, especially their contact information, so the profile pulls from each talent's CV and GitHub while keeping them anonymized.

03

Improved recall

In order to create the conversation design, the LLM's recall needed to be improved to keep track of everything said in the conversation before making the AI talent recommendations. This also allowed the LLM to perform better with follow-up prompts from the user without hallucinating, so this one feature answered two of the problems.

The redesigned flow · two of its screens
Results · three candidates to compare
Results: the request, then three anonymized candidates with a match score, skills and buttons to view a profile or book an interview
Candidate profile · built from the CV and GitHubCandidate profile · from the CV and GitHub
A candidate's profile open beside the results: match score, AI summary, industries and their experience

People compare three candidates in one place, open a full profile, and book an interview without leaving the chat.

Find AI talent · request to booked interviewFind AI talent · request to booking
Ask
The start screen: "what AI talent are you looking for?" with a message box and suggested roles
Profile · use cases, education, skills
The candidate profile scrolled to the candidate's AI use cases, education and skills
Pick a time
Booking an interview with a candidate: a calendar, a call length and time slots
Interview booked
The chat after booking: the candidate is marked Booked and the interview time sits in the conversation

05 · The results

What LyRiseAI Beta showed

After the dev team worked on these improvements, we launched LyRiseAI Beta and tracked some key metrics. These numbers were from about 300 users in the first month.

The average time it took 54 users to find an AI talent match after initiating a chat went down by 54%. 16 of the 54 users that booked an AI talent for an interview responded to our survey, and their confidence in their selection was 19% higher.

Alpha usability tests and the betaAlpha tests and the beta
Chat-only alphaLyRiseAI BetaBars are relative to the alpha

Talent request to match

For the total conversion of initiated chats to talent matches, there was a 37% decrease. My hypothesis for this is that we're comparing data from usability tests (where users are in an interview and given a clear task to try to accomplish) with data from a public beta release where there are many more factors at play (for example, for some users, if the LLM took too long to load, they could have just left). So further testing and analysis will be required to improve this conversion rate.

What I would do differently

In retrospect, I think one of the biggest mistakes made was in the initial user research. It was not prioritized due to the rush to get the product in front of users, which is normally fine as that gets you direct feedback in the shortest time. However, in this case, LLMs were a new, somewhat revolutionary, technology. It would have been wiser to spend more time in the user research phase to better understand how to best implement LLMs for the user's needs.

Next case study
Contact

Get in touch

TTarek

Have a role or project in mind? Send me an email.

Email me