An extensible, state of the art columnar file format. Formerly at @spiraldb, now a Linux Foundation project (@LFAIDataFdn). Apache-2.0
Github
- Tweets 27
- Following 14
- Followers 289
- Likes 67
vortex retweeted
The DAVE stack: the age of domain-optimized analytical databases on Datafusion, Arrow, and Vortex, Embedded
sequenceandsilicon.substack.…
1
10
51
2,032
vortex retweeted
If you happen to be looking for more reasons why you should choose @vortexdotdev as your storage layer:
Check out the QCon London 2026 talk by @onur_satici "From S3 to GPU in One Copy: Rethinking Data Loading for ML Training" spiraldb.com/blog/from-s3-to…
1
1
5
371
❤️ DataFusion
Apache DataFusion is becoming a foundational building block for modern data systems.
From RisingWave, InfluxDB, LanceDB, and GreptimeDB to OpenObserve, ParadeDB, Spice.ai, Vortex, Apache Comet, Ballista, and many others, a growing ecosystem is building on the same shared query foundation.
Why?
Building a high-performance query engine from scratch requires a huge amount of engineering.
A modern engine needs:
SQL and DataFrame APIs
Logical and physical query planning
Query optimization
Vectorized execution
Multithreaded processing
Streaming execution
File-format support
Object-storage integration
Memory management
Extensible functions and operators
Apache DataFusion provides these core capabilities in a modular and extensible engine written in Rust and built around the Apache Arrow in-memory format.
The real value is not just performance.
It is extensibility.
This allows teams to reuse the common foundations of query processing without giving up control over the parts that make their systems unique.
Instead of rebuilding the same SQL parser, optimizer, execution engine, and file-format support again and again, projects can focus their engineering effort on their specific workloads and product differentiation.
RisingWave follows the same approach.
RisingWave uses Apache DataFusion as its default batch query engine while continuing to use its streaming-native engine for incremental and stateful stream processing.
Apache DataFusion handles the query engine, allowing projects to focus on the unique requirements of their specific use cases.
4
375
AI agents and real-time apps increasingly need analytical answers from data that was written seconds ago. Pointing those queries straight at an operational database slows production and widens security risk. The usual alternative, an ETL pipeline into a warehouse, is costly to operate and can lead to data that's stale by hours or days.
Spice 2.0 offers a new approach, and is now available.
It stands up a sandboxed analytic replica beside your operational store and keeps it current with high-throughput change data capture directly from those sources. You point Spice at a database and start running analytical queries in minutes, while production never carries the analytical load.
High-throughput CDC replication is complemented by the rest of the 2.0 platform: multi-node distributed compute on @ApacheDataFusio Ballista for petabyte-scale queries, Spice Cayenne columnar acceleration on the open @vortexdotdev format, a cluster-sidecar architecture that pairs isolated local sidecars with a shared cluster, and enterprise security enforced in the engine with OIDC, Cedar policies, PII masking, and mTLS.
"Spice is the data plane behind Summation. Every connector we ship, from Snowflake to a generic REST-as-a-table, collapses into one SQL surface, which is what makes our AI agents portable across a customer's stack. 2.0's distributed query takes the scaling concern off the table."
-Ramachandra Ramarathinam, CTO at @summation
It's open source, portable, scalable, and fast.
Read the launch blog for more details: hubs.ly/Q04p5pFw0
And, check out the thread ⬇️ for breakdowns of the major features in 2.0.
6
6
11
2,514
vortex retweeted
so cool to see another blazing fast database built on vortex!
Just announced at Interrupt! SmithDB.
Agent traces have outgrown the databases built to hold them.
That’s why we built SmithDB, a purpose-built distributed database for agent observability.
Read the announcement from Co-Founder @ankush_gola11 → langchain.com/blog/introduci…
6
32
71,089
vortex retweeted
We leveraged two amazing open source projects when building SmithDB.
One is @ApacheDataFusio: an extensible Rust based query engine. We built custom execution plans specifically tuned for our workloads and storage backend, and DataFusion made it straightforward to plumb everything together.
The other is @vortexdotdev: an extensible file format that allows you to build custom layouts with specific encoding and chunking strategies for different columns.
I would highly recommend checking out both of these projects if you're interested in modern data systems.
We built SmithDB: the database purpose built for agent observability workloads that now powers many parts of LangSmith.
Agent observability presents a challenging data problem. Agent traces can contain tens of thousands of intermediate spans and large, unbounded payloads. These characteristics are a direct result of agents running for longer time horizons and LLM context window sizes growing.
Traditional data infrastructure was not built to handle the complexities associated with storing and querying this data.
SmithDB brings LangSmith up to 12x performance improvements across access patterns most important for agent observability. I’ve been working on SmithDB directly with an amazing team over the past few months, and I’m incredibly proud of the results we’re seeing.
I wrote a bit more about the story and engineering challenges behind SmithDB in this blog.
Additionally, if you’re a systems engineer interested in building the future of agent observability, please reach out!
2
18
105
18,252
The Research Behind Modern Data Compression & @vortexdotdev
When we chose Vortex as the storage layer for Spice Cayenne (the data accelerator engine in Spice), we were betting on decades of database research finally reaching production-ready maturity.
Here's the research behind Vortex:
📄 BtrBlocks (SIGMOD 2023) - The core algorithm from the Technical University of Munich. Cascading multiple lightweight encodings outperforms monolithic compression. Optimize for decompression speed, not just compression ratio.
📄 FastLanes (VLDB 2023) - Hardware-friendly integer compression. Structures bit-packing to maximize SIMD utilization across AVX-512, AVX2, and ARM NEON. Near-memory-bandwidth decompression.
📄 FSST (VLDB 2020) - Fast Static Symbol Table for strings. Near-LZ4 ratios at 5-10× faster decompression. Critical for string-heavy columns.
📄 ALP (CWI Amsterdam) - Adaptive Lossless floating-Point compression. Exploits real-world float patterns (prices with 2 decimals, sensor readings with limited precision).
📄 MonetDB/X100 + Morsel-Driven Parallelism - Foundations for vectorized, NUMA-aware query execution that Vortex builds on.
The result? Compression that is tailored to your data:
• Integers via FastLanes bit-packing
• Floats via ALP adaptive encoding
• Strings via FSST symbol tables
• Timestamps via delta encoding
• Sorted columns via run-length encoding
Why does this matter for production systems?
1️⃣ Query performance scales with decompression speed. Focus on decode performance translates directly to faster queries.
2️⃣ Automatic encoding selection means zero configuration. The algorithm samples your data and picks optimal strategies per column.
3️⃣ SIMD acceleration is baked in. FastLanes was designed for vectorized, hardware accelerated execution from day one.
4️⃣ Zero-copy Arrow access. Data decompresses directly to Arrow arrays with no intermediate copies.
Vortex is now a Linux Foundation AI & Data project, and researchers are building on it (Anyblox, F3). You get SOTA research in production systems.
The future of data storage is exciting.
To learn more about our Vortex implementation, check out the blog: hubs.ly/Q04bGfvf0
#datafusion #ai #data #vortex #spiceai #arrow #parquet
1
6
414
vortex retweeted
Connor Tsui & I just merged a first cut of TurboQuant into @vortexdotdev , already validated on production embeddings 🚀🚀🚀
1
5
12
2,404
Fastest OSS file format, in both performance and velocity
Connor Tsui & I just merged a first cut of TurboQuant into @vortexdotdev , already validated on production embeddings 🚀🚀🚀
1
316
you took up with Weasley, but he can't afford sliceable cascaded encodings.
now your random access is dogged, and your cortisol is properly spiked, potter
3
128
hey man, thrilled that you're interested in contributing. we'll be waiting for you in slack
vortex.dev/slack
1
192
🦆❤️🚀
DuckDB now supports reading from and writing to the Vortex file format! The DuckDB Labs and Spiral teams have worked together to make Vortex available as a core extension in DuckDB.
Vortex is an open source, columnar file format whose design is heavily influenced by recent research in lightweight compression encodings, computing and IO techniques.
We gave it a test drive, and it performed very well. Read the full article to learn more lnkd.in/eZfGzPiZ
2
6
573
🌪️ Why LF Vortex for hot data?
@ApacheParquet great compression, slow decode
@ApacheArrow instant decode, no compression
Vortex: encoding-efficient compression with SIMD decode to Arrow
80% of Parquet's compression, 10x faster decode
1
5
11
832
vortex retweeted
Happy to share that I've been nominated to the @vortexdotdev Technical Steering Committee! It's been fun and productive switching to Vortex from Parquet as our storage format at Polar Signals and I'm excited to continue contributing to the Vortex project.
1
1
4
362
vortex retweeted
Super cool, they forked @DeltaLakeOSS to replace Parquet (for data) with Vortex and JSON (for metadata) with Vortex. Huge performance gains!
Maybe we should upstream this one 😁 @vortexdotdev
🧊 New on the Polar Signals Blog — Our Delta Lake Fork
Purpose-built for our continuous profiling product. In our latest post, we walk through how Delta Lake works, and the changes we've made to improve performance for our product.
👉 Read the full post: buff.ly/KwHINtO
4
5
65
7,794
vortex retweeted
So cool!! Polar Signals reduced query runtimes by 70% switching from Parquet to Vortex 🤯🚀
We completed a major project to switch our storage file format from Parquet to Vortex 🌪️ resulting in 70% average query performance improvement across the board 🚀
Learn more about how rethinking interface-imposed limitations unlocked these gains in our latest blog post 👇
3
24
2,426
vortex retweeted
We completed a major project to switch our storage file format from Parquet to Vortex 🌪️ resulting in 70% average query performance improvement across the board 🚀
Learn more about how rethinking interface-imposed limitations unlocked these gains in our latest blog post 👇
2
7
27
3,751