multi-agent collaboration

The energy is crazy, DM me or grab me at the @cohere booth if you wanted to chat. Also come see @chxdie and I talk at 3pm in room 7A (PSE 7303/7363) Fac Hall.

a propos of nothing, i'll be at @HackTheNorth this weekend! stop by the @cohere booth and say hi if you can't find me, follow your 👃

Today, more than 100 leading AI experts endorsed a set of minimum requirements to take seriously AI companies' recent call to embed external evaluators. These evaluators need to be genuinely independent, transparent, and represent a range of expertise areas. They also need to be guaranteed employee-level access and to be protected from retaliation for findings that make companies look bad. We welcome model developers’ recent calls for independent oversight, but it’s what they do next that matters. The labs must be accountable for ensuring these requirements are met, so that the public can have faith in the process and the outcomes. Over the past week, the AI community has debated the appropriate role of external evaluation, including who should do it and on what terms. We may not agree on everything, but there is a lot of common ground. To make embedded evaluations credible, more than 100 experts with varying backgrounds and ideas about AI risk agree in today’s letter that frontier AI developers should: 1. Guarantee embedded evaluators full editorial independence and mitigate conflicts of interest 2. Rely on multiple evaluators with differing viewpoints and areas of expertise 3. Publicly document the terms under which evaluators operate, as well as facilitating permissive publication of methods and findings 4. Shield evaluators from retaliation 5. Grant access equivalent to that of highly privileged employees There is a thriving and growing ecosystem of independent AI evaluators who are advancing this science every day – but we need aligned standards, guaranteed protections, and independent funding. That’s why we created the AI Evaluator Forum. Today we are entering our next phase. We’re launching an open call for new members, collaborators, and independent funding sources to help evaluators meet this moment and demand accountability from developers. Join us in building the evaluator ecosystem. See the public letter here: aievaluatorforum.org/initiat… Learn more at aievaluatorforum.org/path-ah…

What is the root cause of this disgusting stench in Waterloo? It started yesterday morning

Someone please give me $450 so I can buy the limited edition lunar new year's Tim Hortons mahjong set

Introducing Benchmark Reviews: our new initiative to audit AI benchmarks. We are launching with 15 benchmarks: 4 Verified, 9 Flawed, and 2 with not enough information for a review.

"Thus, to maintain that sovereignty, we must pursue diminished reliance upon the United States." Kind of a non sequitur -- but I joined @cohere this month. If you're also a Canadian living in SF feeling weird about the current moment, i'm always happy to talk 🙂

Stephen Harper: "It is clear that the current U.S. administration views our level of economic integration as incompatible with our separate sovereignty. Thus, to maintain that sovereignty, we must pursue diminished reliance upon the United States."

2. If you really nail these -- have high standards, strong penalties for not following them, and mechanisms for ensuring it's apparent when you aren't following them -- any needed slowdown takes care of itself to some extent. Maybe it should be explicitly targeted, maybe not.