Quiz : Unsupervised Learning with Clustering

While the objective function is convex, the optimum value is at infinity when the data is perfectly separable. Hence logistic regression need not converge.

To understand this intuitively consider the picture above : There are an infinite number of sigmoids that can separate the same set of points if they are perfectly separable.

Logistic regression will keep trying to go to the ideal step function and the weights keep exploding. In this case, using a regularization term helps.

• Taking a moving average of the previous readings is a reasonable first thing to try, except that it does not capture changes due to external conditions. It is reactive rather than predictive.
• Learning a regression model from readings in past intervals  +eternal factors solves this problem. But time series data typically is auto-correlated, which means the error term in the linear  regression (or the residual) is not independent from that of previous terms
• ARIMA, which stands for Autoregressive Integrated Moving Average, is a model for time series data that incorporates both autoregressive and moving average features, along with de-trending of the data. The AR part means that the values are regressed on their own lagged values, the MA part means that the regression error is a linear combination of past error terms. The I part means that the data have been differenced to remove trend. This is a common model  for time series analysis.
• Recently, LSTMs are being used for time-series models when there is enough training data.
• Principal component analysis(PCA) is a dimensionality reduction technique, not necessarily used for regression.

We need more data to avoid overfitting. But repeating the same data does not necessarily help since it simply has the effect of reducing the regularization weight by giving double the weight to the initial set of points leading to more overfitting. Repeating data does not add new information.

The model has overfit the data leading to a non-linear relationship, learning a more complex model than required to explain the data. To reduce over-fitting one can either increase the regularizer(L2 regularizer in this case) weight or add more data in training.

Eigenvalues of a diagonal matrix are the diagonal elements. In general, eigen values of a matrix A are defined as all those such that for some vector x. Consider vector . Then . Similarly for , . Hence the answer is 2nd option, i.e.,  eigenvalue is 1 and is the corresponding eigenvector.

140 like tea while 80 like both tea and coffee. So 60 like only tea. 120 like coffee and  80 like both tea and coffee, so 40 like only coffee. Then 60 + 40 + 80 = 180 like either tea or coffee. 

K-means is a clustering algorithm while PCA is used for dimensionality reduction (not necessarily clustering). Hence 1 and 3 can be used to find movie clusters while 2nd option does not make sense.

One can get data about movies from many public sources including IMDB, wikimedia and so on. But about a particular customer, we cannot get enough data about his or her preferences(for personalisation) until they start using the application.

Underfitting is the opposite of overfitting and it occurs when model is too simple to fit the data well enough. This could happen if the right features were not selected or extracted, or there is high regularization .

• To avoid underfitting, one way is to make the model more complex, with more parameters for instance.
• If you’re convinced that your model is complex enough, try to increase the number of features or extract new features from the existing ones to solve the underfitting problem.
• Once you’ve performed both the steps, reduce the regularization hyperparameter.

Online learning is also done in offline mode. Here the word online means incremental. So you train the algorithm by feeding the data sequentially. Incremental or online learning is useful when the algorithm needs to adapt to a rapidly changing data, or when resources, such as time for training or computing resources, are limited. Many a times, using entire dataset for training may not be feasible as explained here. 

The bias-variance tradeoff is  a core concept in supervised learning.

We want to design models that best fit the training data capturing all the subtleties in the training data, at the same time generalize well to unseen test data. The bias-variance tradeoff says we cannot do both well simultaneously.

→ If we fit the training data very well, we might end up overfitting to the training data. This might cause high variance in predictions when we try the model on various versions of test data.

→ If we avoid overfitting training data by making the model simple, say, by using a regularizer, we might end up underfitting the training data. We end up with a biased predictor, but it might work well on unseen test data – the variability in predictions across different test data is low (low variance).

Ideally we want low bias (works best on training data), low variance (generalizes well to test data). But we need to pick a tradeoff point.

The naive Baye’s classifier makes the naive assumption that  various feature dimensions are independent conditioned on the class .

Naive Bayes classifier : Suppose you have M dimensional data of the form and you want to predict the class for this data. From Bayes rule we have : 

Using the naive Baye’s assumption of independence of all features  given y, we can write :

Twitter data differs from wikipedia data in a number of ways :

1. Twitter data is very noisy.
   1. Spelling errors
   2. Abbreviations
   3. Code mixing – where multiple languages are used
   4. Grammatical mistakes
2. Tweets are very short
3. There is a lot of variability in the way the language is used, arising from informal and colloquial usages.
4. Many out of vocabulary words that are not in wiki.

Wikipedia data on the other hand is characterized by long well formed sentences that are curated carefully and is much cleaner than tweets data. However, wiki data is crowdsourced, and hence not fully reliable as well.

From the perspective of building word embeddings,  pretrained models, such as word2vec on google news dataset and glove are much readily transferable to wikipedia data than twitter. With twitter data, existing models fall short and embeddings must be trained on tweets.

Note that reliability of data is a less important factor in building word embeddings as long as the co-occurrence characteristics are met. So this is not necessarily a challenge.

If your model is overfitting that means it doesn’t generalize well on the new unseen data. An overfitting model is highly complex as it is trying to fit every example and even the noise in the data. So the prediction error on new data is due to the algorithm complexity. Increasing number of parameters in the model is one way of making it more complex model. 

Bias error arises due to biases we have towards certain kind of data because we can’t access all kind of data, for instance. Due to this bias, there are not right number of features enough for the model to fit accurately.  

Correlation coefficient(Coeff) only measures linear correlations and may completely miss out on non linear relationships. But if Coeff is 0.0, it doesn’t mean there is definitely a nonlinear relationship. Instead, there could be non linear  relationship but Coeff can’t find it.

Note that correlation coefficient can be a powerful tool to examine the relationships between the true labels and any single feature in training data or between any two features as part of feature engineering task.

Time series data is contextual and any shuffling will make the data random and independent, i.e. the relation between adjacent data samples would be lost after shuffling. On the other hand, shuffling should be done while doing train, test and validation split to avoid any biases in dataset or any kind of ordering due to the way data is generated.  For some online algorithms like SGD classifier, shuffling is necessary as their efficiency relies on the randomness in the training set.

Usually traffic on a website is huge with large number of normal requests but anomalies are very less. This leads to large amount of data with very less anomalies. Hence a skewed dataset problem. In any skewed dataset, accuracy measure should not be used. This is because, even if we learn a very bad classifier predicting all examples as the majority class, we would get a good accuracy score – but this does not mean we want to pick this classifier.

The F1 score based on precision and recall and the AUC metric are better metrics on an imbalanced dataset.

When building an ML model, often one needs to trade off between

• Bias and variance trade-off : Trade-off between how well the model fits the existing training data vs how well we want the model to generalize to new data)
• Precision and recall : Recall measures how many of all the positives are actually classified positive. Precision measures how many of the items identified as positive are actually positive. It is easy to get a recall of 1 or a precision of 1, but hard to get high precision and recall – we want to pick the option that leads to the highest combination of precision and recall. F1 score is a metric to measure this.
• MLE and MAP : MLE stands for the maximum likelihood estimate – the value that maximises the likelihood function. MAP refers to the Maximum a posterior estimate – the value that maximises the posterior distribution. There is no trade-off here as in the ideal case, with infinite data, both estimates converge.

Cross-validation involves partitioning  training data into complementary subsets, performing the analysis on one subset (called the training set), and validating the analysis on the other subset (validation set). The validation set is rotated between various partitions to reduce variance of the estimated model.

With a very small dataset, when K is  small, say 2, the training set has only half the available data, leading to the possibility of over-fitting. A higher K, such as 10, ensures 90 data samples during training which is better than just 50 samples. Hence, for this question with just 100 data points, a higher value of K makes sense.

With a larger data-set, a small value of K might suffice, if each fold has enough data points. With a lower  K in such a case, the training time might be lower.

Matrix multiplication is associative, i.e. (AB)C = A(BC)
Matrix multiplication is distributive, i.e. A(B+C) = AB+AC
Matrix multiplication may not be commutative, i.e. AB = BA only if m=q in the given cas.e

|x| is  a linear function which has a minimum at x=0. If you plot the graph of |x|, you’d realise it has a unique minimum at x=0. Also by property of convex function, it should be clear that |x| is convex function. Note that convex function definition have nothing to do with function differentiability.

Exponential function on open interval is convex.

Apply the convex function property to realise that this is a convex function.

What it means is a monotonically non decreasing function can be convex too.

Note that double differentiation of this function is f”(x) =

f”(x) >= 0 for all values of x because of

Strictly convex function means the function has not more than one minimum.

This function has only one minimum which is at x = 0 as f'(x) = = 0 at x=0

Preprocessing is done after train test split. Note that the purpose of train test split is to ensure better generalization. If test set is also included in preprocessing or preprocessing is done on entire dataset, the purpose test set is lost which was to mimic new unseen data. Hence first data is split into train and test set then preprocessing is done on train set.

Feature engineering includes feature selection and feature extraction. Features are nothing but the attributes of each data point. Feature engineering is about the entire problem set and requires domain expertise. It is not much dependent on dataset but more on the problem in hand. Data preprocessing is performed on entire training set. If during data preprocessing one figures out two features are correlated, one can get single feature from both features by combining them. Hence both are different tasks as feature engineering is on feature(selection and extraction) and data preprocessing is on the dataset by removing or filling missing values for each feature, removing irrelevant features etc.

Training set, test set and validation set all must have same features. Only difference is data preprocessing is done(fit) on training set(combined with validation set) and applied(transform) on test set. Actual steps involved are

1. Split dataset into train and test set
2. Preprocessing on training set
3. Split training set into training and validation sets
4. Transform test set using same preprocessing done in (2)
5. Evaluate model on transformed test set

Though test set is used for testing generalization but it may not have same number of points. Standard is to keep 20% of entire dataset for test set but even this can change depending on the total number of points. For eg, if there are more than million samples, even 10% can be good for test set but if there are 10,000 samples, 20% is good for test set.

All tasks are important for the success of any ML algorithm. Unless we know what problem in hand or how is the dataset, it is difficult to know what steps can be skipped. 

True labels should be correlated to some features. If they’re not correlated we may not be able to predict correct values from the features. Only due to some correlation between true labels and features, supervised learning models are able to predict correct value. This correlation can be either linear or nonlinear.

Private variables of a class with an underscore by convention. Hence local variables do not usually start with an underscore. Refer to the PEP guide on coding conventions for more details.  https://www.python.org/dev/peps/pep-0008/

Python core library does not contain an array. It has a list that is a generalized array of dynamic length and capable of holding dissimilar elements. Numpy contains an array that is optimized for performance in terms of space and speed. All the remaining choices are built-in types in python.

The map function executes a function given as the first argument on all the elements of the list (iterable) given as the second argument. Check out the following link to learn more about map https://www.geeksforgeeks.org/python-map-function/.

There is no great way to deal with missing data but use many heuristics such as those mentioned above.

1. The most common way of dealing with missing data is to remove all rows with missing data if there are not too many rows with missing data.
2. If more than 50-60% of rows of a specific column are missing data, it is common to remove the column. The main problem with removing missing data thus, is that it could  introduce substantial bias.
3. Imputation of data is also a common technique used  to deal with missing data where the data is substituted with the best guess.
   1. Imputation with mean : Missing data is replaced by mean of the column
   2. Imputation with median : Missing data is replaced by mean of the column
   3. Imputation with Mode: Missing data is replaced with mode of the column
   4. Imputation with linear regression : With real valued data, this is a common technique. The missing value is replaced by performing linear  regression based on the other feature values.

Python is more prevalent as the defacto language for data science. Python has better ML libraries and visualization tools.

However Scala is more efficient to use on spark. Scala uses Java Virtual Machine (JVM) during runtime which gives it some speed over Python in most cases. Python is dynamically typed and this reduces the speed. Compiled languages are faster than interpreted.

• Python is dynamically typed (unlike Java, C, C++).  It doesn’t know about the type of the variable until the code is run and the binding happens on the go.
• Python is an interpreted language. This means it uses an interpreter and is not compiled directly to machine code. Python is compiled to byte code and an interpreter executes the statements one-by-one from the code, on the go. In compiled languages like c++ on the other hand,  the compiler executes the entire code and shows all errors before creating a binary.
• Python has automatic garbage collection unlike c++ where badly written programs can lead to memory leaks. 

The model is overfitting to the training data possibly leading to low error on the training data, however the model is not able to generalize well to the test data.

Note that the other choice of “underfitting” is wrong, since underfitting the training data would have lead to high training error.

A fundamental aspect of Machine Learning is the ability to learn from the available training data and “generalize” to new unseen test data.

The curse of dimensionality is a core concept in Machine Learning. As the number of dimensions grows (i.e number of features), we need exponentially more data to fit the model.

1. As the number of data points grows, more computational effort is much more to search in a higher dimensional space through more parameter combinations
2. If we try to increase the dimensionality of the function we are trying to fit,  data points in the training data, are sparser in the higher dimensional space than the lower dimensional space (average distance between points increases). So more data is needed.

A common technique to combat this is dimensionality reduction, to remove redundant dimensions.

Look at this article for more information https://medium.freecodecamp.org/the-curse-of-dimensionality-how-we-can-save-big-data-from-itself-d9fa0f872335

Train data is typically used for training the model.

Validation data is used for hyper parameter tuning. Usually validation data is a part of the train data. Often k-fold cross validation is used where 1 in k parts of the training data is used at a time for validation.

Test data The test data is usually held out data with gold standard labels available. Once hyperparameters are learnt on the validation data, a final metric is computed on the testing data based on which a decision can be taken whether the model is acceptable.

F1 score is the only classification metric. Note that logistic regression is a classification algorithm. All the others metrics Mean absolute error, Root mean square error, Goodness of fit metric are regression metrics.

For more information on F1  score and classification metrics, refer to :

https://mli-test2-24c419.ingress-baronn.easywp.com/topics/machine-learning/you-have-come-up-with-a-spam-classifier-how-do-you-measure-accuracy/

The K-means algorithm has two steps :

• For each point compute closest cluster center. If there are K clusters and N data points, D dimensions each, this step takes O(N*K*D)
• For each cluster, compute new cluster mean : O(N*K*D) overall
• If there are I iterations, we repeat this process I times making the complexity O(K*N*D*I).

For more information on the k-means clustering algorithm, please refer to :
https://www.datascience.com/blog/k-means-clustering

Dropout ensures some of the hidden units are dropped out at random to ensure the network does not overfit by becoming too reliant on a neuron by letting it overfit.
Note that dropout is often considered a form of regularization to ensure the network does not memorize the target during learning.
For more information on Dropouts – see here.

Both SVM and KNN can fit non-linear decision boundaries. Note that SVM can do arbitrarily complex decision boundaries with the kernel trick. KNN can also fit non-linear decision boundaries by playing with the K value.

SVM is relatively fast during prediction time (depending on the kernel used). The prediction runtime is O(num of support vectors * number of features) since the process involves taking the dot product of the datapoint (for which you want to predict) with each support vector.

Prediction with KNN could be slow. Because Computing the K nearest neighbours for a new data-point depends on the specific data structure used (A Kd-tree is a common data structure used) and. However the prediction complexity still has a log(n) term where n is the number of data points. For huge datasets KNN does not work well.
Note that recently approximate KNN techniques based on Locality Sensitive Hashing are also commonly used.

Training error is high : You probably built a bad model: You have underfit at least some portion of your training data (it is possible you overfit some other portions simultaneously leading to high variance! )

You can :

• Make the model more complex by increasing the number of parameters (this could be number of neurons/hidden layers in a deep learning model)
• Use more features to make the model more complex
• Clean up your data: Your data could be inconsistent and full of outliers that is causing a bad fit. Cleaning the data might result in a better model.
• Decrease the regularizer if you used very high regularization

The naive Baye’s classifier makes the naive assumption that various feature dimensions are independent for a given class.

Naive Bayes classifier : Suppose you have M dimensional data of the form and you want to predict the class for this data.

From Bayes rule we have :

Using the naive Baye’s assumption of independence of all features given y, we can write :

Each of the values of , and and are parameters of the algorithm that can be learnt. Once these parameters are learnt, for a new data point, we can use the formula above to make a prediction.

In python, when two arrays have a different shape, the smaller array is “broadcast”  across the larger array. In the above example a is broadcast to

array([[0, 1, 2],
       [0, 1, 2],
       [0, 1, 2]])

b is broadcast to

array([[0, 0, 0],
       [1, 1, 1],
       [2, 2, 2]])

The resulting vector is the following :

array([[0, 1, 2],
       [1, 2, 3],
       [2, 3, 4]])

Note the following rules of broadcasting : (https://docs.scipy.org/doc/numpy-1.13.0/user/basics.broadcasting.html)

When operating on two arrays, NumPy compares their shapes element-wise. It starts with the trailing dimensions, and works its way forward. Two dimensions are compatible when

1. they are equal, or
2. one of them is 1

Keras provides many high level abstractions making it easy to get going with deep learning models with minimal code. However, debugging code is hard since it is harder to locate the actual line of code that breaks, since one cannot examine the values of intermediate results in the deep learning model.

Pytorch on the other hand, while more verbose to build models, is much more convenient to debug.

Check out https://deepsense.ai/keras-or-pytorch/

Spark is a framework for performing general data analytics on top of a distributed computing cluster like Hadoop. Note that spark leverages HDFS for the underlying file system and does not have its own file system support. It provides in memory computations for increased speed and data processing over Hadoop’s Mapreduce framework.

The key difference between Hadoop MapReduce and Spark is that Spark can do processing in-memory, while Hadoop MapReduce has to read from and write to a disk. As a result, the speed of processing is better with Spark. For the same reason, MapReduce is typically more suited for batch processing.

A VM runs a unique version of the OS, complete with memory management. Multiple VMs only share hardware resources.

Docker containers are typically more lightweight compared to VMs. A container does not have its own memory management layer. A kernel bug can be exploited from an app running in the container on the host OS. Data in another container can be read/ changed if such a  kernel bug allows access to contents of another container.

Further,  Docker images are usually smaller than VM images, makes it easy to build, copy, share. Second, Docker containers can start in several milliseconds, while VM starts in seconds.

Equality operator in python makes both variables refer to the same object. It does not copy. To copy an object in python, one can do a deep copy or a shallow copy.

Shallow copy creates a new object and copies references to the constituents of the original object to the new object. For instance if you have a list of lists, shallow copying it will create a new top level list, but the constituent lists will still refer to the list objects in the original object.  list.copy() does a shallow copy.

Deep copy not only makes a copy of the top level object, but recursively deep copies all constituent objects. For more details see the article below :  

https://medium.com/@thawsitt/assignment-vs-shallow-copy-vs-deep-copy-in-python-f70c2f0ebd86

Pandas is a wrapper over Numpy to support working with tabular data easily such as column names, SQL like query support, merging and joining dataframes. 

Numpy supports highly optimized multidimensional numeric arrays for high performance computing. For performance optimized  ML code, NumPy is  often preferred over Pandas. 

Lists in Python are a native data type.  Lists are dynamic  in size and need not necessarily contain homogeneous elements.  Numpy arrays are homogeneously typed and are not built over Lists.  

Analyzing which variables correlate more with the target variable that we are trying to predict is a standard technique in supervised ML: The pearson correlation coefficient is the most commonly used measure of correlation – it ranges between  -1 and 1.

• A highly positive or negative value indicates a strong relationship
  • Features that are highly correlated with target are important.
  • If two features are very highly correlated with each other, one of them might be redundant. Example Age and salary might be highly correlated. Age+2 and salary are also likely to be correlated !  
• A value close to 0 between a feature and target indicates very little relationship
  • Such features can be removed reduce dimension of data
  • Derived features can be computed which are better correlated with target to replace features that show very little correlation. Example:  You are predicting whether a person is claiming unemployed allowance. The salary of the person might be correlated with the unemployment allowance received. But a derived feature (salary>0) that is indicative of employment is a better predictor.
• Note: The pearson’s correlation coefficient captures only the linear correlation.

In Online transaction processing (OLTP), systems typically facilitate and manage transaction-oriented applications, where they need to retrieve information and update the database in an atomic fashion for each transaction.

Hive is not suited for OLTP systems since it does not support insert and update at the row level.

For more information on Hive, take a look at https://cwiki.apache.org/confluence/display/Hive/Home

MinMax Scaling results into values bounded between 0 and 1. It is performed by subtracting all values with the minimum value and dividing the result by the maximum value minus minimum value, i.e.
processed_value = (original_value – minimum_value) / (maximum_value – minimum_value). Standardization by subtracting the mean and dividing by the variance results into 0 mean but unbounded values.

Categorical features are usually present in textual format. For eg, three categories in a size column could be LOW, MEDIUM, HIGH. These categories if converted straightaway to 0,1,2 within the same column, the machine learning model will learn assuming there is some ordering as 0 < 1 < 2. Though in example there is ordering but in reality categories may not have any such ordering. To avoid such problems, one-hot encoding is used. In one-hot encoding, one column is added for each category. And the original column with all categories is removed. Each column takes value of 0 or 1 for each row depending upon whether that row has that category or not.

Complexity of SVM algorithm can be quadratic in the size of training set. This is especially true for kernel SVM due to the quadratic size of kernel matrix. Note that it is possible most of them will take more time as the number of examples are increased, but we need to answer out of all the models mentioned, which one will scale poorly, relatively.

For any Machine Learning task in hand, we need to first give some time to understand the problem. Sometimes it is done with the help of domain experts. Once the problem has been understood, we need to check if it is a supervised or unsupervised learning problem. This can be determined by knowing what is the desired goal and whether we have labels or not. Once we know if its a clustering or classification problem, we do the train test split for better generalization. In most cases it is train-validation-test split but for simplicity we kept train-test split. Notice that train-test split happens before the data preprocessing as we don’t want test data to be used in preprocessing for better generalization. Whatever preprocessing is applied(sklearn’s fit function) on training data, test data is transformed similarly(using sklearn’s transform function). Post train-test split , we select the right features and then pre-process the data by scaling, normalization, replacing missing values in the features and other required processing in this step. And then lastly, we select the right model. Note that model selection would also include the task of determining what loss function to use.

Categorical features are usually present in textual format. For eg, three categories in a size column could be LOW, MEDIUM, HIGH. These categories if converted straightaway to 0,1,2 within the same column, the machine learning model will learn assuming there is some ordering as 0 < 1 < 2. Though in example there is ordering but in reality categories may not have any such ordering. To avoid such problems, one-hot encoding is used. In one-hot encoding, one column is added for each category and hence extra K column for K categories. And the original column with all categories is removed leading to D + K – 1 features. Each column takes value of 0 or 1 for each row depending upon whether that row has that category or not.