Can a single neural network policy generalize over poses, objects, obstacles, backgrounds, scene arrangements, in-hand objects, and start/goal states? Introducing Neural MP: A generalist policy for solving motion planning tasks in the real world 🤖 1/N

Very excited to show the Demo of our proposed ReconX, a novel 3D scene reconstruction paradigm that reframes the ambiguous reconstruction challenge as a temporal generation task. Project page: liuff19.github.io/ReconX/ #3dgs #aigc

Box CEO Aaron Levie on why AI agents are a $1 trillion opportunity "AI will not just be serving back an answer for you, it will also go and do something for you ... as tokens come down in cost, we can scale up intellectual labor and knowledge work [with AI] orders of magnitude more than we could have with people." - @levie with @TurnerNovak on @ThePeelPod

"Spann3R: 3D Reconstruction with Spatial Memory" In a nutshell: DUSt3R strikes again! Paper: arxiv.org/abs/2408.16061 Project: hengyiwang.github.io/project… Method ⬇️

It's a tradition for hackers to run DOOM in crazy places: thermostats, "smart" toasters, even ATMs. Now they run DOOM purely in a diffusion model. Every pixel here is generated. A while ago, I said "Sora was a data-driven physics engine". Well, not quite, because Sora could not be interacted with. You set the initial condition (a text or initial frame) and may only watch the simulation passively. GameNGen is a proper neural world model. It takes as input past frames (states) and an action from a user (keyboard/mouse), and outputs the next frame. The quality is by far the most impressive I've seen on DOOM. However, this comes with significant caveats. Let's deep dive: 1. GameNGen overfits to the extreme on a single game by training on 0.9B frames (!!). This is a HUGE number, almost 40% of the dataset used to train Stable Diffusion v1. At this point, it's likely memorizing how DOOM renders from every corner of the game in all scenarios. DOOM doesn't have that much content anyway. 2. GameNGen is more like a glorified NeRF than a video gen model. A NeRF takes images of a scene from different view angles, and reconstructs the 3D representation of the scene. The vanilla formulation has no generalization capability, i.e. it could not "imagine" new scenes. GameNGen is not like Sora: by design, it could not synthesize new games or interaction mechanics. 3. The hard part of this paper is not the diffusion model, but the dataset. Authors trained RL agents to play the game first, at various different skill levels, and collected 0.9B (frame, action) pairs for training. Most of the video datasets online do NOT come with actions, which means this method wouldn't extrapolate. Data is always the bottleneck for action-driven world models. 4. There're two practical use cases for game world models in my mind: (1) write a prompt to create playable worlds that would otherwise take game studios years to make; (2) use the world model to train better embodied AI. Neither use cases can be realized. Use case (2) doesn't work because there's no advantage to use GameNGen for training agents than directly using DOOM simulator itself. It'd be more interesting if a neural world model simulates scenes that traditional hand-crafted graphics engines cannot. What's an example of a truly useful neural world model? @elonmusk said in a reply that "Tesla can do something similar with real world video". Not surprising: Autopilot team likely has trillions of (camera feed, steering wheel action) pairs. Again, data is the hard part! With such rich real-world data, it's entire possible to learn a general driving sim that covers all kinds of edge cases, and use that to deploy & verify a new FSD build without physical cars. GameNGen is still a really great proof of concept. At least we know by now that 0.9B frames is the upper bound to compress high-res DOOM into a neural network.

One line of code. 20X faster data processing. Watch @RAPIDSai cuDF accelerate pandas code processing 7.2M rows of telecom data in an interactive viz in seconds. Try out the notebook on Colab: nvda.ws/3X4T2fA

Claude 3.5 Sonnet transformed a research paper into an interactive learning dashboard in just 30 seconds on my iPhone. It goes beyond the capabilities of GPT-4o, Gemini Pro, Llama and other existing LLMs. Education will never be the same again with AI.

i made 5 bodyguard agents in minecraft... and then made them fight to the death for my affection

here's a quick look at an internal tool we made to build agents faster. we iterate on our agent's internal structure, run through dynamically generated prompts, and test new samples of outputs more in-distribution of how our agents act. let us know below if you'd like access

This paper is wild. Create a camera with cheap terrible camera lens, but train a diffusion model to recreate a much better image. arxiv.org/abs/2408.07541

This is so awesome, Verba is on GitHub Explore! ✨ I'm super happy to see our open-source RAG shining on the explore page; big thanks to the @github Team for featuring it. ❤️ Let’s go open source! GitHub Explore: github.com/explore Verba GitHub: github.com/weaviate/verba

Introducing Omni Engineer 🚀 An AI coding framework built for control and effectiveness. Create and edit multiple files in REAL time Web search via DuckDuckGo Save and resume conversations Image support Use any model you want via @openrouter Here's a quick demo 👨‍💻

🤖 Introducing @Taskade's AI Teams with Multi-Agent Collaboration, now in beta! Assemble your team of AI Agents for collaboration, intelligent responses, and dynamic interactions. They plan, execute, and manage tasks—all together! Reply 'AI Teams' for early access! ✨

llama 3.1 70b on groq

ROSA (the #ROS Agent) is finally open-source! ROSA is an AI agent designed to interact with ROS-based robotics systems using natural language queries.

Vehicle analytics zone + object counting using @ultralytics🚓🔥 🔗Code & docs: docs.ultralytics.com/guides/… New tools and technologies are emerging every day, enhancing the precision and user-friendliness of traffic analytics.💙 #ai #counting #vehicles

Grounded SAM 2: Ground and Track Anything in Videos github.com/IDEA-Research/Gro… Based on the core concept of Ground SAM, we have introduced Grounded SAM 2. We have open-sourced our code, allowing users to input custom videos and create impressive object tracking videos. #AI #SAM2

3D Gaussians Splats look good, but they are less interesting when static. Our ECCV 2024 work, Feature Splatting, enhances 3DGS with foundational vision features and physical simulation to synthesize dynamic scenes from static 3D captures. ✨Open-sourced in NerfStudio!👇

Explore how NVIDIA NIM microservices are advancing #generativeAI in speech AI, retrieval, digital biology, digital humans, simulation, and #LLMs. Read the technical blog ➡️ nvda.ws/3LX7Wj0.