Week 5 : Stanford cs231 and KNN

So Ive started with this Convnets course on Stanford University.

I watched the first three videos. Turned out to be just the thing that I am looking for. They also have some homeworks and all, so I will be able to test myself and put some milestones on the way.

Best part of it is the fact that one of the objectives that we have to accomplish as a part of this course is gaining the ability code our own convolutional neural networks from scratch. Thats encouraging.

To start with the first assignment, I downloaded Anaconda. It is a tool to manage your different python installations. It makes it possible to manage them in different environments which you can activate and deactivate. I shouldda discover it a long time ago.

In assignments, Stanford provides some requirement.txt s that give us the exact state of the environment that our code needs to run. to create an environment and to use it is fairly easy. you just have to go

conda create --name myenv

and

source activate myenv

after that. If you cant remember the name of your environment, you can simply go

conda info --envs 

---

Lets get started with the course. After listening the first 3 videos, it will be rather easy to cope with knn homework. With is the first part of the first assignment. In my experience, one skill I had to gain was the vectorization. Which can be a bit tricky to comprehend as a concept if you're not familiar with it.

It all starts with the concept of broadcasting, which enables us to do arithmetical operations between matrices with unequal dimensions.

<code>

Python 2.7.14 |Anaconda custom (64-bit)| (default, Oct 16 2017, 17:29:19)
[GCC 7.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> a = [1,2,3,4,5,6,7]
>>> b=4
>>> a*b
[1, 2, 3, 4, 5, 6, 7, 1, 2, 3, 4, 5, 6, 7, 1, 2, 3, 4, 5, 6, 7, 1, 2, 3, 4, 5, 6, 7]
>>> import numpy as np
>>> np.dot(a,b)
array([ 4,  8, 12, 16, 20, 24, 28])

</code>

as can be seen in the example above, as a result of the broadcasting capabilities of numpy, it is possible to have the dot product of a constant and a matrix. What really happens is the multiplexed evaluation of that single constant. In other words, the evaluator will interpret our statement as

a = [1,2,3,4,5,6,7]
b = [4,4,4,4,4,4,4]

Another example, which has been used by Andrew Ng in Coursera 

<code>
>>> print(A)
[[ 56.    0.    4.4  68. ]
 [  1.2 104.   52.    8. ]
 [  1.8 135.   99.    0.9]]
>>> cal = A.sum(axis=0)
>>> print(cal)
[ 59.  239.  155.4  76.9]
>>> print(A/cal.reshape(1,4))
[[0.94915254 0.         0.02831403 0.88426528]
 [0.02033898 0.43514644 0.33462033 0.10403121]
 [0.03050847 0.56485356 0.63706564 0.01170351]]
>>> print(A/cal.T)
[[0.94915254 0.         0.02831403 0.88426528]
 [0.02033898 0.43514644 0.33462033 0.10403121]
 [0.03050847 0.56485356 0.63706564 0.01170351]]
>>> print(A/cal)
[[0.94915254 0.         0.02831403 0.88426528]
 [0.02033898 0.43514644 0.33462033 0.10403121]
 [0.03050847 0.56485356 0.63706564 0.01170351]]
</code>

But, what does this has to do with the vectorization?

Its gets better.

Consider you have two huge matrices that carry your test and training data. 5000 images for training and another 500 for testing. And you like to get the sum of the distances of every single image peer of this [Test]x[Training] cartesian. First thing comes to mind : Write a nested loop and consider it done. But it will takes extremely long to finish them all?

Yep.

Lets try to understand it in a simpler way.
Lets say that our images are basically constants e.g. 4,56,7,8 etc.
and our huge matrices are 3 and 2 inner matrices long instead of 5000 and 500. and the resulting loopy code will be sth like that

<code>
a = np.array( [[1,2,3,4,5],[1,2,3,4,5],[1,2,3,4,5]] )
b = np.array( [[6,7,8,9,10],[6,7,8,9,10]] )

sum =0
for i in a:
  for j in b:
      sum = sum + abs(a[i]-b[j])

</code>

what about the vectorized code?

sum = np.sum(np.abs(a-b[:,np.newaxis]))




---

[1] https://conda.io/docs/user-guide/tasks/manage-environments.html