4 ms·

How would you go about generating such dataset? 1. Scrape images and store as png 2. Downscale to 28px 3. Convert each image to grayscale 4. Convert to matr

by nip 9y ago

How would you go about generating such dataset? 1. Scrape images and store as png 2. Downscale to 28px 3. Convert each image to grayscale 4. Convert to matrices and add label (additional row?) 5. Normalize to have matrices of 1 and 0 for faster computation 6. Vectorize said matrices 7. Concatenate into one big vector Did I miss something / Am I fooling myself? I plan on working on my first ML side project and I would love to gain some insights from HN.

e_ameisen 9y ago
That's the right idea overall, with a few caveats. 1. Yes but you need to manually inspect and verify that the images are of the right class 5. Images are grayscale, not only black and white. Additionally, MNIST and fashion-MNIST have all their objects centered and of similar scale. This is a large part of what makes them a popular first test for any image model: they are very simple to solve as the model need not be very robust to fit the dataset.
- Q6T46nT668w6i3m 9y ago
  > Additionally, MNIST and fashion-MNIST have all their objects centered and of similar scale. This is a large part of what makes them a popular first test for any image model: they are very simple to solve as the model need not be very robust to fit the dataset. Yeah, this is crucial. Especially when trying to generalize models. It’s easy to verify the usefulness of data augmentation when you can make basic assumptions about the data (e.g. it’s centered).
Q6T46nT668w6i3m 9y ago
4. It’s usually preferable if your dependent (target) variable is separate from your independent (output) variable for usability reasons. 5. I’d provide a vector of integers where each integer represents a different class. Encoding is dependent on the underlying algorithm and its implementation. You’d want to provide plenty of flexibility to users. I’d also add that MNIST has an advantage over a dataset like CIFAR because background pixels are zeroed out (i.e. you don’t need to account for varied backgrounds). So you’d probably want to segment your objects.
evolve2k 9y ago
This might help. Their academic paper is basically a description of how they made the dataset (and why) and includes a great example of the steps they went through to convert the images. https://arxiv.org/pdf/1708.07747.pdf https://arxiv.org/pdf/1708.07747.pdf

by nip 9y ago

e_ameisen 9y ago

That's the right idea overall, with a few caveats. 1. Yes but you need to manually inspect and verify that the images are of the right class 5. Images are grayscale, not only black and white. Additionally, MNIST and fashion-MNIST have all their objects centered and of similar scale. This is a large part of what makes them a popular first test for any image model: they are very simple to solve as the model need not be very robust to fit the dataset.

Q6T46nT668w6i3m 9y ago
> Additionally, MNIST and fashion-MNIST have all their objects centered and of similar scale. This is a large part of what makes them a popular first test for any image model: they are very simple to solve as the model need not be very robust to fit the dataset. Yeah, this is crucial. Especially when trying to generalize models. It’s easy to verify the usefulness of data augmentation when you can make basic assumptions about the data (e.g. it’s centered).

Q6T46nT668w6i3m 9y ago

4. It’s usually preferable if your dependent (target) variable is separate from your independent (output) variable for usability reasons. 5. I’d provide a vector of integers where each integer represents a different class. Encoding is dependent on the underlying algorithm and its implementation. You’d want to provide plenty of flexibility to users. I’d also add that MNIST has an advantage over a dataset like CIFAR because background pixels are zeroed out (i.e. you don’t need to account for varied backgrounds). So you’d probably want to segment your objects.

evolve2k 9y ago

This might help. Their academic paper is basically a description of how they made the dataset (and why) and includes a great example of the steps they went through to convert the images. https://arxiv.org/pdf/1708.07747.pdf https://arxiv.org/pdf/1708.07747.pdf