Hi Reader, In case you missed it, I launched a free, 7-hour pandas course! 800+ students have enrolled, and a few have already earned their certificate of completion ๐ฉโ๐ ๐ Link of the weekโData Internshipsโ Looking for an internship in Data Science or Analytics? This site curates the latest internship postings and emails them to you each week! ๐ Tip #40: Build a DataFrame from multiple filesLetโs say that your dataset is spread across multiple files, but you want to read the dataset into a single pandas DataFrame. For example, I have a tiny dataset of stock market data in which each CSV file only includes a single day. Hereโs the first day: Hereโs the second day: And hereโs the third day: You could read each CSV file into its own DataFrame, combine them together, and then delete the original DataFrames, but that would be memory inefficient and require a lot of code. A better solution is to use Pythonโs built-in glob module: You can pass a pattern to the In this case, glob returns filenames in an arbitrary order, which is why we sorted the list using Pythonโs built-in We can then use a generator expression to read each of the files using Unfortunately, there are now duplicate values in the index. To avoid that, we can tell the Pretty cool, right? Need to build a DataFrame column-wise instead? Use the same code as above, except pass axis='columns' to ๐ Until next timeDid you like this weekโs tip? Please forward it to a friend or share this link in your favorite Slack team. It really helps me out! ๐ See you next Tuesday! - Kevin P.S. Would you wear pajamas during a Zoom call?โ Did someone AWESOME forward you this email? Sign up here to receive Data Science tips every week! |
Join 25,000+ aspiring Data Scientists and receive Python & Data Science tips every Tuesday!
Hi Reader, I'm really proud of this week's tip because it covers a topic (data leakage) that took me years to fully understand. ๐ง It's one of those times when I feel like I'm truly contributing to the collective wisdom by distilling complex ideas into an approachable format. ๐ก You can read the tip below ๐ or on my blog. ๐ Link of the week Building an AI Coach to Help Tame My Monkey Mind (Eugene Yan) In this short post, Eugene describes his experiences calling an LLM on the phone for coaching:...
Hi Reader, Last week, I recorded the FINAL 28 LESSONS ๐ for my upcoming course, Master Machine Learning with scikit-learn. That's why you didn't hear from me last week! ๐ I edited one of those 28 videos and posted it on YouTube. That video is today's tip, which I'll tell you about below! ๐ Tip #45: How to read the scikit-learn documentation In order to become truly proficient with scikit-learn, you need to be able to read the documentation. In this video lesson, Iโll walk you through the five...
Hi Reader, happy Tuesday! My recent tips have been rather lengthy, so I'm going to mix it up with some shorter tips (like today's). Let me know what you think! ๐ฌ ๐ Link of the week A stealth attack came close to compromising the world's computers (The Economist) If you haven't heard about the recent "xz Utils backdoor", it's an absolutely fascinating/terrifying story! In short, a hacker (or team of hackers) spent years gaining the trust of an open-source project by making helpful...