Open source ML and LLM evaluation 📊 , testing 🚦and monitoring 📈 GitHub: github.com/evidentlyai/evide… Discord: discord.gg/xZjKRaNp8b
- Tweets 2,259
- Following 211
- Followers 2,547
- Likes 1,206
Pinned Tweet
3️⃣ 2️⃣ 1️⃣ Our free course on LLM evaluations for AI product teams starts today!
🎥 7 days of byte-sized videos into your inbox
⭐️ Certificate upon completion
👩💻 No coding skills required
👩🎓500+ students have signed up
You can still join the course👇
evidentlyai.com/llm-evaluati…
3
1
7
1,908
How DoorDash runs evals for support chatbots using simulated conversations - at scale: careersatdoordash.com/blog/d…
1
2
207
How Netflix evaluates the quality of show synopsis using LLM-as-a-judge approach:
netflixtechblog.com/evaluati…
1
4
1,077
How Zalando builds a search quality assurance framework with LLM-as-a-judge:
engineering.zalando.com/post…
144
📌 In case you missed it
How to evaluate an AI agent?
Follow the tutorial as we:
1️⃣ Build an AI agent,
2️⃣ Create a test dataset,
3️⃣ Assess responses and tool choice,
4️⃣ Track the agent’s behaviour.
Follow the tutorial from our LLM evals course: piped.video/watch?v=9KMmadw7…
1
223
A Friday ML use case 📕
📚 From the database of 800 ML & LLM systems: cutt.ly/SwrZWL0g
How Uber improves driver availability at airports: Estimated time-to-request model, Earnings-per-hour prediction, and Driver-deficit forecasting.
uber.com/en-GB/blog/forecast…
157
🦾 More AI agents aren’t always better.
Google evaluated 180 agent setups and found multi-agent systems help with parallel tasks but can hurt sequential ones.
The work also proposes a model to predict optimal agentic designs.
research.google/blog/towards…
1
110
📌 In case you missed it
Let’s test your RAG system!
Follow the tutorial as we:
1️⃣ Build a RAG system,
2️⃣ Generate test data,
3️⃣ Evaluate answers for correctness and faithfulness.
Watch the tutorial from our LLM evals course: piped.video/watch?v=jckp5R09…
140
A Friday ML use case 📕
📚 From the database of 800 ML & LLM systems: cutt.ly/SwrZWL0g
How GoDaddy built Lighthouse, an internal AI analytics platform: prompt engineering framework, model orchestration, solution architecture, and use cases.
godaddy.com/resources/news/h…
87
Evidently AI retweeted
(policyNIM oss tool)
preflight command is working. when I provide a coding task, it kicks off a search through indexed policies to determine which rules are relevant for implementation.
@nvidia for embedding w/ @OpenAI + @lancedb for vector storage.
eval command is also working. using @EvidentlyAI for running eval suite.
1
2
4
446
🚦 Meta’s “Agents Rule of Two”
According to Meta, AI agents should satisfy at most two of these conditions per session to reduce prompt-injection risk:
- Handle untrusted inputs
- Access sensitive data
- Change state / act externally
ai.meta.com/blog/practical-a…
58
📌 In case you missed it
How do you know if your RAG works?
You need to check:
✅ Can it find the right information?
✅ Is the final answer complete, relevant, and free of hallucinations?
Watch the intro to RAG evaluation from our LLM evals course: piped.video/watch?v=qI2qQfOG…
1
177
A Friday ML use case 📕
📚 From the database of 800 ML & LLM systems: cutt.ly/SwrZWL0g
How DoorDash improves its RecSys using LLMs to bridge behavioral silos in multi-vertical recommendations.
careersatdoordash.com/blog/d…
68
💭 Can AI systems introspect?
Anthropic’s new research suggests Claude models can sometimes identify and describe their own internal states.
It’s still unreliable, but marks a step toward more transparent AI reasoning.
anthropic.com/research/intro…
53
📌 In case you missed it
Can LLMs write engaging tech tweets?
Follow the tutorial as we:
1️⃣ Build a tweet generator,
2️⃣ Score its outputs with custom LLM judges,
3️⃣ Improve the results with prompt iteration.
Watch the tutorial from our LLM evals course: piped.video/watch?v=KhkiM9C0…
2
186
A Friday ML use case 📕
📚 From the database of 800 ML & LLM systems: cutt.ly/SwrZWL0g
How Shopify transformed its product classification system from basic categorization to an AI-driven framework using Vision Language Models.
shopify.engineering/evolutio…
1
2
76
📚 Context is everything.
OpenAI shares how it built an in-house data agent that answers complex questions in minutes.
It uses 6 layers of context:
- Table metadata
- Human annotations
- Codex enrichment
- Company knowledge
- Memory
- Runtime context
openai.com/index/inside-our-…
1
118
📌 In case you missed it
Are LLMs good for classification tasks?
We built an LLM-based classifier for a travel support chatbot and compared its performance to a classic ML model.
Watch the tutorial from our LLM evals course: piped.video/watch?v=Gl2X_o99…
1
2
164
A Friday ML use case 📕
📚 From the database of 800 ML & LLM systems: cutt.ly/SwrZWL0g
How Wayfair built Wilma, a customer service agent copilot: workflow, prompt templates, and the copilot’s evolution.
aboutwayfair.com/careers/tec…
88
🤖 How to develop and deploy chatbots at scale?
DoorDash shares how they created a simulation platform and evaluation flywheel, allowing them to test chatbots with fast feedback loops and without production risk.
careersatdoordash.com/blog/d…
1
1
65
📌 In case you missed it
How to create an LLM judge that aligns with human labels:
- Define criteria
- Create test dataset
- Run evaluation prompt to see if the judge aligns with your labels
- Evaluate the judge
Watch the video from our LLM evals course: piped.video/watch?v=kP_aaFnX…
1
1
177