Thinking..
12:09 PMBased on the job description, I have identified three suitable candidates:Thanks for your patience. Here are 3 anonymized candidates that match your ML engineer role.
Candidate A is an ML Engineer with 8 years of experience, based in Beirut (GMT+3). Machine learning engineer with experience on NLP tasks, working in TensorFlow, PyTorch, Flask, PostgreSQL and Elasticsearch. Relevant use cases include product recommendations.
Candidate AML Engineer8 yearsBeirut (GMT+3)96%matchMachine learning engineer with experience on NLP tasks, working in TensorFlow, PyTorch, Flask, PostgreSQL and Elasticsearch.
Relevant use casespProduct recommendations+4ShortlistView profileBook interviewCandidate B is an ML Engineer with 6 years of experience, based in Beirut (GMT+3). 6+ years in machine learning with a focus on NLP, including text classification and summarization. Knows TensorFlow, Keras, PyTorch and BERT. Relevant use cases include product recommendations.
Candidate BML Engineer6 yearsBeirut (GMT+3)86%match6+ years in machine learning with a focus on NLP, including text classification and summarization. Knows TensorFlow, Keras, PyTorch and BERT.
Relevant use casespProduct recommendations+4ShortlistView profileBook interviewCandidate C is an ML Engineer with 5 years of experience, based in Beirut (GMT+3). Computer vision background with production models for imaging interpretation. Less direct work on recommendation systems than the role asks for. Relevant use cases include imaging interpretation.
Candidate CML Engineer5 yearsBeirut (GMT+3)81%matchComputer vision background with production models for imaging interpretation. Less direct work on recommendation systems than the role asks for.
Relevant use casesiImaging interpretation+4ShortlistView profileBook interviewThinking..
12:09 PMBased on the job description, I have identified three suitable candidates:Thanks for your patience. Here are 3 anonymized candidates that match your ML engineer role.
Candidate A is an ML Engineer with 8 years of experience, based in Beirut (GMT+3). Machine learning engineer with experience on NLP tasks, working in TensorFlow, PyTorch, Flask, PostgreSQL and Elasticsearch. Relevant use cases include product recommendations.
Candidate AML Engineer8 yearsBeirut (GMT+3)96%matchMachine learning engineer with experience on NLP tasks, working in TensorFlow, PyTorch, Flask, PostgreSQL and Elasticsearch.
Relevant use casespProduct recommendations+4ShortlistView profileBook interviewCandidate B is an ML Engineer with 6 years of experience, based in Beirut (GMT+3). 6+ years in machine learning with a focus on NLP, including text classification and summarization. Knows TensorFlow, Keras, PyTorch and BERT. Relevant use cases include product recommendations.
Candidate BML Engineer6 yearsBeirut (GMT+3)86%match6+ years in machine learning with a focus on NLP, including text classification and summarization. Knows TensorFlow, Keras, PyTorch and BERT.
Relevant use casespProduct recommendations+4ShortlistView profileBook interviewCandidate C is an ML Engineer with 5 years of experience, based in Beirut (GMT+3). Computer vision background with production models for imaging interpretation. Less direct work on recommendation systems than the role asks for. Relevant use cases include imaging interpretation.
Candidate CML Engineer5 yearsBeirut (GMT+3)81%matchComputer vision background with production models for imaging interpretation. Less direct work on recommendation systems than the role asks for.
Relevant use casesiImaging interpretation+4ShortlistView profileBook interviewBeforeChat-only alpha
AfterTalent profiles
54%Limited betafaster to a match
LyRise · AI Talent Matcher · 2023
From chat replies
to confident choices.
LyRise's first AI matcher answered in plain text, and employers could not tell which candidate to pick. I redesigned it around talent profiles. In the limited beta, time to a match was 54% lower than on the chat-only alpha.
- Role
- Product designer, full-time, reporting to the CEO
- Company
- LyRise, AI startup
- Year
- 2023
- Tools
- Figma
Objective
Improve the conversion from talent request initiation to successful talent interview booking.
- 54%decrease in time to match
- 19%higher confidence in selection
- ~300users in the limited beta in its first month
01 · Context
Rebuilding the platform around LLMs
LyRise is a platform for matching employers in the US and EU with AI talent in Africa and South-east Asia. ChatGPT and other LLMs had just launched, so we were rebuilding our platform with the integration of LLMs.
In an agile startup, we adopted a mentality of always putting users first, and putting our product in front of users as soon as possible. So I created an MVP feature list that I extrapolated from our core function's user flow, and our dev team developed an alpha version with minimal UI, a prototype almost.

MVP feature list
Must have
- LyRiseAI LLM chat interface
- Talent database
- Talent shortlisting
- Interview booking
Delighters
- Chat to job request auto fill
- Full talent profiles (viewed by asking the LLM for more info about a talent)
02 · Problem discovery
Testing the alpha with users
After the alpha version was developed, I started testing the product with users. I conducted:
28 usability tests
I conducted 28 remote usability tests. The objective of the usability test was to have a user initiate a chat with the LLM to find an AI talent for a pre-defined role. I wanted to see:
- how users interacted with the LLM
- how long it took them to find an AI talent they would want to interview
- what steps they would like to see next

28 surveys
For those same users from the usability tests, I would fill out a post-usability test survey to document some qualitative and quantitative data. Some of those were:
- whether the user found an AI talent they want to interview or not
- if they did, how confident were they in their selection?
- if they didn't, why didn't they?
LyRiseAI Alpha post-usability test survey
This survey is filled by the UX researcher for each usability test, the form itself is updated as per usability test notes.
Was the user able to successfully find an AI talent they were interested in interviewing?
If yes, how long did it take?
Short answer textIf yes, how confident is the user in their selection?
If no, why?
Did the LLM hallucinate at any point?
If yes, after how many prompts did the LLM begin hallucinating?
Short answer text16 user interviews
A few months earlier, I had conducted 16 user interviews with our main target persona to understand their process for hiring and what criteria they assess when looking for an AI talent for an AI role.

User interview questions bank
User interview answers
Could you briefly go over your hiring process?
We typically start with a detailed job description outlining the specific AI skills and experience needed for the role. We then post the job on various platforms and review incoming resumes. Shortlisted candidates go through a technical screening, followed by multiple rounds of interviews with the head of HR, me (CTO) and sometimes the CEO as well.
What is the most painful stage in that process?
Sourcing qualified AI talent for sure. There's a huge gap between the number of AI roles and qualified candidates, so it's incredibly time-consuming to find candidates with the right mix of skills and experience.
What criteria do you have when reviewing applicant profiles?
I usually look for relevant experience in AI, specific skills for the required role. I also review their GitHub repositories, this is the most important factor for me. Cultural fit, problem solving abilities and potential for growth are also important factors.
A CTO, in one of the interviews
The feedback from these activities also served in finding the solutions.
03 · The problems
Three key problems
Usability tests and user interviews showed us we had three key problems:
01
Time consuming
Users were taking too long to figure out how to properly prompt the LLM to recommend AI talent. By my estimates, it would take the user roughly 3 to 4 minutes just trying to figure out how to prompt the LLM.
02
Low confidence
Users didn't feel too confident in the recommended AI talent because they didn't have enough information until they asked the LLM for more info.
03
Low recall
When users asked a chain of follow-up questions to an original prompt, the model would begin hallucinating by the third or fourth follow-up at best.
04 · The solutions
Three key features
To address these issues we implemented three key features:
01
Conversation design
The goal of conversation design is to decrease the unpredictability of user prompts by having the LLM guide the user by asking specific questions, and once the LLM has sufficient info, it will make a recommendation.
02
Full talent profile
The most common reason for low confidence in users' selection was lack of access to the talent's full CV and experience. So I, together with the dev team, worked on creating a full talent profile that includes all the criteria I gathered from user interviews with our target persona a few months earlier. Talents had to stay anonymous, especially their contact information, so the profile pulls from each talent's CV and GitHub while keeping them anonymized.
03
Improved recall
In order to create the conversation design, the LLM's recall needed to be improved to keep track of everything said in the conversation before making the AI talent recommendations. This also allowed the LLM to perform better with follow-up prompts from the user without hallucinating, so this one feature answered two of the problems.


People compare three candidates in one place, open a full profile, and book an interview without leaving the chat.




05 · The results
What LyRiseAI Beta showed
After the dev team worked on these improvements, we launched LyRiseAI Beta and tracked some key metrics. These numbers were from about 300 users in the first month.
The average time it took 54 users to find an AI talent match after initiating a chat went down by 54%. 16 of the 54 users that booked an AI talent for an interview responded to our survey, and their confidence in their selection was 19% higher.
Talent request to match
For the total conversion of initiated chats to talent matches, there was a 37% decrease. My hypothesis for this is that we're comparing data from usability tests (where users are in an interview and given a clear task to try to accomplish) with data from a public beta release where there are many more factors at play (for example, for some users, if the LLM took too long to load, they could have just left). So further testing and analysis will be required to improve this conversion rate.
What I would do differently
In retrospect, I think one of the biggest mistakes made was in the initial user research. It was not prioritized due to the rush to get the product in front of users, which is normally fine as that gets you direct feedback in the shortest time. However, in this case, LLMs were a new, somewhat revolutionary, technology. It would have been wiser to spend more time in the user research phase to better understand how to best implement LLMs for the user's needs.
Have a role or project in mind? Send me an email.