Blazing Fast I/O of Data in the Cloud

I/O from remote storage is a consistent bottleneck for large scale data processing workloads. When you have hundreds of thousands of files in S3 storage, even listing those files can take several minutes and become a bottleneck! Reading those files can be even more painful than the actual processing of the files.Daft Dataframes are built for the cloud and feature many optimizations that make them extremely efficient at reading and working with cloud storage. In this talk, we will showcase and explain some of the optimization that are built into Daft using its Rust I/O layer, but exposed to users as a familiar Python Dataframe interface.