Tuesday Tip #40: Build your DataFrame from multiple files ๐Ÿ—๏ธ


Hi Reader,

In case you missed it, I launched a free, 7-hour pandas course!

800+ students have enrolled, and a few have already earned their certificate of completion ๐Ÿ‘ฉโ€๐ŸŽ“


๐Ÿ”— Link of the week

โ€‹Data Internshipsโ€‹

Looking for an internship in Data Science or Analytics? This site curates the latest internship postings and emails them to you each week!


๐Ÿ‘‰ Tip #40: Build a DataFrame from multiple files

Letโ€™s say that your dataset is spread across multiple files, but you want to read the dataset into a single pandas DataFrame.

For example, I have a tiny dataset of stock market data in which each CSV file only includes a single day. Hereโ€™s the first day:

Hereโ€™s the second day:

And hereโ€™s the third day:

You could read each CSV file into its own DataFrame, combine them together, and then delete the original DataFrames, but that would be memory inefficient and require a lot of code.

A better solution is to use Pythonโ€™s built-in glob module:

You can pass a pattern to the glob() function, including wildcard characters, and it will return a list of all files that match that pattern.

In this case, glob() is looking in the โ€œdataโ€ subdirectory for all CSV files that start with the word โ€œstocksโ€ followed by one or more characters:

glob returns filenames in an arbitrary order, which is why we sorted the list using Pythonโ€™s built-in sorted() function.

We can then use a generator expression to read each of the files using read_csv() and pass the results to the concat() function, which will concatenate the rows into a single DataFrame:

Unfortunately, there are now duplicate values in the index. To avoid that, we can tell the concat() function to ignore the index and instead use the default integer index:

Pretty cool, right?

Need to build a DataFrame column-wise instead? Use the same code as above, except pass axis='columns' to concat()!


๐Ÿ‘‹ Until next time

Did you like this weekโ€™s tip? Please forward it to a friend or share this link in your favorite Slack team. It really helps me out! ๐Ÿ™Œ

See you next Tuesday!

- Kevin

P.S. Would you wear pajamas during a Zoom call?โ€‹

Did someone AWESOME forward you this email? Sign up here to receive Data Science tips every week!

Learn Artificial Intelligence from Data School ๐Ÿค–

Join 25,000+ intelligent readers and receive AI tips every Tuesday!

Read more from Learn Artificial Intelligence from Data School ๐Ÿค–

Hi Reader, This week, I've got a short tip about AI agents, followed by some Data School news... ๐Ÿ‘‰ Tip #56: What are AI agents? Google is calling 2025 "the agentic era," DeepLearning.AI says "the agentic era is upon us," and NVIDIA's founder says "one of the most important things happening in the world of enterprise is agentic AI." Clearly AI agents are a big deal, but what exactly are they? Simply put, an AI agent is an application that uses a Large Language Model (LLM) to control its...

Hi Reader, Last week, I launched a brand new course: Build an AI chatbot with Python. 120+ people enrolled, and a few have already completed the course! ๐Ÿ‘ Want to join us for $9? ๐Ÿ‘‰ Tip #55: Should you still learn to code in 2025? Youโ€™ve probably heard that Large Language Models (LLMs) are excellent at writing code: They are competitive with the best human coders. They can create a full web application from a single prompt. LLM-powered tools like Cursor and Copilot can autocomplete or even...

Hi Reader, The Python 14-Day Challenge starts tomorrow! Hope to see you there ๐Ÿคž ๐Ÿ‘‰ Tuesday Tip: My top 5 sources for keeping up with AI I'll state the obvious: AI is moving incredibly FAST ๐Ÿ’จ Here are the best sources I follow to keep up with the most important developments in Artificial Intelligence: The Neuron (daily newsletter) My top recommendation for a general audience. Itโ€™s fun, informative, and well-written. It includes links to the latest AI news and tools, but the real goldmine is...