Top 7 Python Libraries for Large-Scale Data Processing
This article covers Python libraries that make large-scale data processing faster, more scalable, and easier to manage across modern data workflows.
dask
Building End-to-End Data Pipelines with Dask - KDnuggets
Learn how to implement a parallelization process in your data pipeline.
Processing a Directory of CSVs Too Big for Memory with Dask - KDnuggets
Learn how to process the big data in the CSV format.
An Introduction to Dask: The Python Data Scientist's Power Tool - KDnuggets
Ever wondered how to handle large data without slowing down your computer? Let’s learn about Dask, a tool that helps you work with large data quickly.
Lessons Learned from Scaling to Multi-Terabyte Datasets
This post is meant to guide you through some of the lessons I’ve learned while working with multi-terabyte datasets. The lessons shared are focused on what someone may face as the size of the…
Beyond Numpy and Pandas: Unlocking the Potential of Lesser-Known Python Lib
3 Python libraries for scientific computation you should know as a data professional.
If You Can Write Functions, You Can Use Dask
This article is the second article of an ongoing series on using Dask in practice. Each article in this series will be simple enough for beginners, but provide useful tips for real work. The first article in the series is about using LocalCluster.
Dask DataFrames — How to Run Pandas in Parallel With Ease
Are you a Data Scientist experienced with Pandas? Then you know its pain points. There's an easy solution - Dask - which enables you to run Pandas computations in parallel.
What is Dask and How Does it Work?
This article will first address what makes Dask special and then explain in more detail how Dask works. So: what makes Dask special? Python has a rich ecosystem of data science libraries including…
State of the art NLP at scale with RAPIDS, HuggingFace and Dask
See how to build end-to-end NLP pipelines in a fast and scalable way on GPUs — from feature engineering to inference.
Are You Still Using Pandas to Process Big Data in 2021?
Pandas doesn’t handle well Big Data. These two libraries do! Which one is better? Faster?
Pandas on the Cloud with Dask
Scaling your Pythonic data science and machine learning to the cloud using Dask. All from the comfort of your own laptop.
DASK: A Guide to Process Large Datasets using Parallelization
A simple solution for data analytics for big data parallelizing computation in Numpy, Pandas, and Scikit-Learn Frameworks.
Pandas with Dask, For an Ultra-Fast Notebook
Use Pandas with Dask to save time and resources. This combination will make your notebook ultra fast
Why and How to Use Dask with Big Data
The Pandas library for Python is a game-changer for data preparation. But, when the data gets big, really big, then your computer needs more help to efficiency handle all that data. Learn more about how to use Dask and follow a demo to scale up your Pandas to work with…
Set up a Dask Cluster for Distributed Machine Learning
Create a Dask environment only by connecting machines using SSH