profile

Learn Data Science from Data School ๐Ÿ“Š

Tuesday Tip #40: Build your DataFrame from multiple files ๐Ÿ—๏ธ

Published about 2 months agoย โ€ขย 1 min read

Hi Reader,

In case you missed it, I launched a free, 7-hour pandas course!

800+ students have enrolled, and a few have already earned their certificate of completion ๐Ÿ‘ฉโ€๐ŸŽ“


๐Ÿ”— Link of the week

โ€‹Data Internshipsโ€‹

Looking for an internship in Data Science or Analytics? This site curates the latest internship postings and emails them to you each week!


๐Ÿ‘‰ Tip #40: Build a DataFrame from multiple files

Letโ€™s say that your dataset is spread across multiple files, but you want to read the dataset into a single pandas DataFrame.

For example, I have a tiny dataset of stock market data in which each CSV file only includes a single day. Hereโ€™s the first day:

Hereโ€™s the second day:

And hereโ€™s the third day:

You could read each CSV file into its own DataFrame, combine them together, and then delete the original DataFrames, but that would be memory inefficient and require a lot of code.

A better solution is to use Pythonโ€™s built-in glob module:

You can pass a pattern to the glob() function, including wildcard characters, and it will return a list of all files that match that pattern.

In this case, glob() is looking in the โ€œdataโ€ subdirectory for all CSV files that start with the word โ€œstocksโ€ followed by one or more characters:

glob returns filenames in an arbitrary order, which is why we sorted the list using Pythonโ€™s built-in sorted() function.

We can then use a generator expression to read each of the files using read_csv() and pass the results to the concat() function, which will concatenate the rows into a single DataFrame:

Unfortunately, there are now duplicate values in the index. To avoid that, we can tell the concat() function to ignore the index and instead use the default integer index:

Pretty cool, right?

Need to build a DataFrame column-wise instead? Use the same code as above, except pass axis='columns' to concat()!


๐Ÿ‘‹ Until next time

Did you like this weekโ€™s tip? Please forward it to a friend or share this link in your favorite Slack team. It really helps me out! ๐Ÿ™Œ

See you next Tuesday!

- Kevin

P.S. Would you wear pajamas during a Zoom call?โ€‹

Did someone AWESOME forward you this email? Sign up here to receive Data Science tips every week!

Learn Data Science from Data School ๐Ÿ“Š

Kevin Markham

Join 25,000+ aspiring Data Scientists and receive Python & Data Science tips every Tuesday!

Read more from Learn Data Science from Data School ๐Ÿ“Š

Hi Reader, happy Tuesday! My recent tips have been rather lengthy, so I'm going to mix it up with some shorter tips (like today's). Let me know what you think! ๐Ÿ’ฌ ๐Ÿ”— Link of the week A stealth attack came close to compromising the world's computers (The Economist) If you haven't heard about the recent "xz Utils backdoor", it's an absolutely fascinating/terrifying story! In short, a hacker (or team of hackers) spent years gaining the trust of an open-source project by making helpful...

13 days agoย โ€ขย 1 min read

Hi Reader, Today's tip is drawn directly from my upcoming course, Master Machine Learning with scikit-learn. You can read the tip below or watch it as a video! If you're interested in receiving more free lessons from the course (which won't be included in Tuesday Tips), you can join the waitlist by clicking here: Yes, I want more free lessons! ๐Ÿ‘‰ Tip #43: Should you discretize continuous features for Machine Learning? Let's say that you're working on a supervised Machine Learning problem, and...

20 days agoย โ€ขย 2 min read

Hi Reader, I'm so excited to share this week's tip with you! It has been in my head for months, but I finally put it in writing โœ๏ธ It's longer than usual, so if you prefer, you can read it as a blog post instead: Jupyter & IPython terminology explained ๐Ÿ”— Link of the week Python Problem-Solving Bootcamp (April 1-21) Want to improve your Python skills quickly? There's no better way than solving problems, reviewing alternative solutions, and exchanging ideas with others. That's the idea behind...

about 1 month agoย โ€ขย 3 min read
Share this post