“Sooooo…this is kinda garbage,” I told
@medialyst founder
@elvissun and graded his AI-generated media list with an F (27/100). Then I signed on as his company’s Founding Strategic Advisor.
I've been hand-building media lists for 14 years. They each take days to research, vet, and refine. Now Medialyst can generate one in seconds. Even though the output has a failing Carly Score™ right now, we’re moving in the right direction. In July we were at 27.9/100. Elvis made changes and this week’s score is already up 5.8 points.
I’ll be grading a fresh list output every Thursday and publish the number whether it's flattering or not.
HOW WE CALCULATE THE CARLY SCORE™
We test:
- Crossover between journalists I found and what the tool finds x .40
- Accuracy of the emails it returns x .30
- Whether I'd keep or cut the names on the list x .30
All weights add up to 1, no hidden multiplier.
THE BASELINE
July 20 run of a list for a job site’s Stress-Free Six-Figure Careers brief:
- The tool found 7 of the 43 journalists on my hand-built list
- But it only got 4 of their emails right
- Of the 50 total names it returned, I'd keep 7
Score: 27.9/100 = GRADE F
THE FIX
- The tool now hunts for more of a journalist's published work before scoring or admits "not enough evidence" rather than inventing a score
- Author index pages no longer count as articles someone wrote
- Emails and LinkedIn profiles that match a completely different name are no longer treated as confirmed matches
THE RESULT
Same brief rerun on August 6:
- Same crossover journalists score: 7/43
- Same correct emails score: 4/43
- Better kept journalists score: 7/21
Score: 33.7/100. GRADE: Still an F… but 5.8 points better because the tool spit out less trash.
VALIDATION
I sat with Elvis and walked through the most recent list line by line. For the first time the tool's scores matched my gut. It ranked reporters
@Andy Medici at 95 (he actually covered the story),
@Bryan Robinson at 95 (planning on covering in August),
@Jennifer Mattson at 78 (who previously covered a similar report for us), and
@Mary Cunningham at 55 (which is exactly where I’d put her for this story).
MOVING FORWARD
What's still broken:
- The tool surfaced CPA Practice Advisor for a consumer story
- Finding better emails for contacts
- A site I couldn't even confirm was a real publication
- Coursera, which is a program-review site
These misses all get addressed before the next test.
From now on, we’ll be posting a new Carly Score™ every Thursday.
Link to the live scorecard if you wanna check it out below.