kaggle Top8% (681th of 8802) 🥉

Summary of Santander-Customer-Transaction-Prediction

 

kaggle Top8% (681th of 8802) 🥉

useful

Smote+XGboost [Public Score= 0.75161]

• Target 반응변수 비율이 0.1밖에 되지 않아 SmoteOver Sampling 적용후 xgboost 모델링

XGboost [Public Score = 0.88411]

• Smote 성능이 낮아 Xgboost 모델링만 시도

XGboost_tuning [Public Score = 0.89692]

• max_depth(최대 트리깊이)= 3 -> 2, colsample_bytree(트리마다 변수선택비율)=1 -> 0.3, learning_rate(학습률)=0.05 ->0.02

Lgboost [Public Score = 0.89766]

• Lgboost가 XGgboost에 비해 매우 빠르고, 그로인해 parameter tuninig이 용이함

Ensemble Models(XGboost + Lgboost) [Public Score = 0.90043]

• 비선형 모델NuSVC에 StandardScaler 적용

Lgboost_oof [Public Score = 0.90043]

• Out of fold로 Public Score 0.003 상승

Lgboost_oof_augment [Public Score = 0.90060]

• Augment(데이터 증식)을 통해 Public Score 0.00017, cv score 0.00061 상승

Lgboost_oof_frequency [Public Score = 0.90119]

• Frequency변수 추가로 약0.005의 cv score가 증가했지만 publi는 미세하게 증가하였다.

1% solution[private = 0.92159]

• remove fake from test
• concat train and test
• frequency encoding
• 200 model train and predict(LGB)

try

• Sum, std, mean, min, max, skew, kurtosis, etc for row
• Augment
• Random over sampling ,SMOTE over samplint
• Round1, Round2, Round3
• Frequency encoding – solutio이었지만 fake test set을 발견하지 못 함
• var_0 – (var_1, var_2, … var_199), var_1 – (var_0, var_2, … var_199)을 반복하며 time series 변수 찾아봄
• Binning 을 이용해 catecory 적용
• Eda를 통해 useless해 보이는 컬럼제거
• Feature importance top5변수의 상호작용(+, x, etc) 컬럼
• PCA
• Clustering
• NN model ensamble

Learning

• OOF(Out Of Fold)

  

• Agument

  It means that there is no interaction between variables (the variables are completely independent). Here are some examples. If the variables did have interactions, then you might need var_0=5 AND var_1=2.5 AND var_2=15 for target=1.If only one of those 3 occurred you would have target=0 but if all three occurred, you would have target=1. That's interaction. If there was interaction, you cannot train on shuffled columns because you would destroy the interactions that are present in the training data and your model could not learn them. But with this Santander data, there are no interactions, so you can shuffle columns and make new data.

• Fake data

  Removing fake samples is the key of this competition

• Frequency Encoding

  Frequency Encoding works even if continuous variable

• how did adding these 200 new/duplicated features improve the score in a tree model (LightGBM)?

  It turns out LGBM benefits from helping it find interactions too.Instead of waiting for a decision tree to send all the observations with count=1 to the left and count>1to the right. You just give it a new feature with all the count=1 values removed (converted to NAN). Instead of waiting for the decision tree to send count <= 2 to the left and count>2 to the right. You just give it a new feature with all the count<=2 values removed. (Additionally adding new columns like var_x times var_x_FE also helps LGBM find interactions)