Gathering Data

Friday, September 26th

Today, I researched about what datasets to use for my research project, and using a few AI tools to help me, I found quite a few.

However, not all datasets are equal in structure and quality, so I had to program algorithms that manipulated the folder structure of the data, and I did so using Python, and specifically, the pathlib and shutil to move files around into different folders. For example:

from pathlib import Path
import shutil

data = Path("E:\\Material Data\\Dataset #1\\raw")
for folder in data.iterdir():
    if folder.is_dir():
        for volume in folder.iterdir():
            try:
                shutil.move(str(volume), str("C:\\Users\\Brandon\\material-design\\data"))
            except:
                continue

This code snippet transfers iterates through all the files volume from every directory inside the file path data, specifically the E:\\Material Data\\Dataset #1\\raw path and transferred all of the files volume) to the file path C:\\Users\\Brandon\\material-design\\data.

The reason the “from” file path is in a removeable drive E:/ is because currently, I doing this project on a laptop, so I didn’t want to take up additional space on my hard drive in my laptop, because it will slow it down.

That’s all for now, see you next week!