The Cost of Training an AI Model

Friday, October 31st

Yes, it’s Halloween!

Before I went out to my parties, I double-checked my budget and if it’s possible (or not possible) if I could pay (or within a reasonable budget ask) for Colab compute units, which enabled me to use Google’s resources, one of which is their GPUs.

So far, I had been using the A100, which, although powerful, burns though compute units compared to a more modest but lesser-resource-intensive GPU, the T4. I experimented with both drivers, and even though the T4 is relatively slower in training my models compared to the A100, it took up less compute units, which meant that the T4 would give me more bang for my buck: Colab compute units cost $1 for every 10 units, and the A100 took about 7.5 units per hour!

Thus, I made the decision to use the T4 over the A100, despite the A100 being more powerful.

Also, the models trained on the T4 vs the A100 improve in accuracy more quicker, but it may just be a coincidence…

Saturday, November 1st

I tried training my model with more data today, but since I got too desperate for more data, I took in a considerable amount of data that turned out to be bad quality because it wasn’t “in scope”. By that, I mean that some of the data I introduced to my model had unrelated data that threw off my model’s learning so to say, which decreased its performance.

I guess you can say I trained with more "“tainted” data: you can’t beat quality with quantity.