How to troubleshoot pandas scikit-learn multidimensional scaling runs forever

Vote count:

0

I'm trying to visualize data in clusters using multidimensional scaling. I've created a DataFrame and that has dimensions of 1000 rows and 1964 columns. When I try to perform multidimensional scaling on the data the process runs forever. Curiously, I can't seem to end the process by doing a ctrl+c.

Through a process of trial and error I have discovered something magical about the 64th row of the dataset. If I run the process on 63 rows the whole thing is done in a couple seconds. If I bump that up to 64 rows, though, it will never end.

I'm really at a loss as to how I should even approach troubleshooting this. I went through the 1964 columns looking for differences between row 63 and row 64 hoping to find a strange value or something but nothing jumped out at me. Any other way I can get an idea of why 64 rows is so magical?

import pandas as pd
from pandas import DataFrame as df
from sklearn.metrics.pairwise import euclidean_distances
from sklearn.metrics.pairwise import manhattan_distances
from sklearn import manifold
from matplotlib import pyplot as plt
import prettyplotlib as ppl

malware_df = df.from_csv('malware_features.csv')

plottable = malware_df[[c for c in malware_df.columns if c != 'hash']]
plottable = plottable.head(63) # change this to 64 and everything stops working

euc = euclidean_distances(plottable)
mds = manifold.MDS(n_jobs=-1, random_state=1337, dissimilarity='precomputed')

plottable['xpos'] = pos_e[:,0]
plottable['ypos'] = pos_e[:,1]
with ppl.pretty:
    fig, ax = ppl.subplots(figsize=(6,8))
ppl.scatter(ax, plottable.xpos, plottable.ypos)
plt.show()

Here is a link where you can download the file I'm using if that helps.http://ift.tt/1t6kWC0

asked 58 secs ago

Kevin Thompson

573

How to troubleshoot pandas scikit-learn multidimensional scaling runs forever