Long's homepage

Practical Data Science

2019-11-15T00:00:00+00:00

My latest tutorial on “Practical Data Science”. R programming language and Jupiter notebooks are used in this tutorial. However, the concepts are generic and can be applied for Python or other programming language users as well.

In this tutorial, I have discussed many essential steps in data science projects, including importing data, data manupulation, visualization, modeling and reporting. For each step, important libraries and example code are provided so that you can quickly reused them for your projects. For example:

Importing data: readr, data.table, RMySQL
Data Manipulation: dplyr, tidyr, lubridate, stringr
Data Visualization: ggplot2, plotly
Data modeling: caret, lm, randomForest, rpart
Reporting: Jupyter notebook, RMarkdown

The data modeling section starts with an overview on predictive modeling landscape, then goes through frequently used models (glm, randomforest, gbm, nerural net). Advanced ML models are also discussed with tips and tricks for tunning hyper parameters. Finally, the stacking technique which is used in many data science competitions is also covered.

Here is the deck:

And, the notebooks:

Hope it useful.

Practical Data Science was originally published by Long H. Nguyen at Long's homepage on November 15, 2019.

Roadtrip along Highway 1 to Big Sur

2019-07-04T00:00:00+00:00

Highway 1 is the best-known scenic drive in Monterey County, and one of the top scenic drives in the world. It’s a National Scenic Byways Program All-American Road and a State Scenic Highway. It’s been named one of the best scenic drives in the U.S. by many famous magazines, such as Forbes, SmarterTravel, Lonely Planet, and etc.

Bixby bridge - The best known and most photographed iconic of the Big Sur.

Partington Cove’s history comes alive as you walk through the 60 foot tunnel and pop out in a secluded cove where rusty remains of the loading apparatus perch on the rocks.

Roadtrip along Highway 1 to Big Sur was originally published by Long H. Nguyen at Long's homepage on July 04, 2019.

Bash Scripting for Hive

2018-08-13T00:00:00+00:00

This post will provide basics about bash scripts, quick guide for running Hive in command lines, and essential tips with example for running Hive script on servers with input parameters and loop.

Bash Scripting Basic

Builtin shell variables

$0             #Name of this shell script itself.
$1             #Value of first command line parameter (similarly $2, $3, etc)
$#             #In a shell script, the number of command line parameters.
$*             #All of the command line parameters.
$-             #Options given to the shell.
$?             #Return the exit status of the last command.
$$             #Process id of script (really id of the shell running the script)

IF and CASE condition statements

if [ "$VAR1" = "$VAR2" ]; then
	echo "expression evaluated as true"
else
	echo "expression evaluated as false"
fi

case "$C" in
"1")
	do_this()
	;;
"2" | "3")
	do_what_you_are_supposed_to_do()
	;;
*)    #otherwise
	do_nothing()
	;;
esac

FOR loop

for i in 1 2 3 4 5    # can also be written for i in {1..5} or {start..end..increment}
do
	echo "Welcome $i times"
done

WHILE loop

while [ condition ]
do
	command
done

Hive CLI (Command Line Interface)

Run Query

hive -e 'select a.col from tab1 a'

Run Non-Interactive Script

hive -f script.sql

Run script inside shell

source file_name

Setting Configuration Property for current Hive

hive --hiveconf year_mm=201808 -f script.sql

Run a Hive Script Even after You Logout

Nohup is very helpful when you have to execute a shell-script or command that take a long time to finish. In that case, you don’t want to be connected to the shell and waiting for the command to complete. Instead, execute it with nohup, exit the shell and continue with your other work.

nohup hive --hiveconf year_mm=201808 -f script.sql

By default, the standard output will be redirected to nohup.out file in the current directory. And the standard error will be redirected to stdout, thus it will also go to nohup.out. So, your nohup.out will contain both standard output and error messages from the script that you’ve executed using nohup command.

Bash Scripting for Hive

Suppose that your Hive script needs to loop over several months (or years) and requires two parameters, year_mm and pre_year_mm. The Hive script is named as my_hive_script.hql

Below bash script can help to execute your Hive script over the months and notify you by sending an email when the job is done.

EMAIL="nlong@gmail.com"

echo "CMI: Finding common accounts within same months of two years ...."

yyyymm=(201808 201807 201806 201805 201804 201803 201802)
prev_yyyymm=(201807 201806 201805 201804 201803 201802 201801)
for((i=0;i<${#yyyymm[@]};i++))
do
  echo "Running for month: ${yyyymm[i]}, previous month: ${prev_yyyymm[i]}"
  hive --hiveconf year_mm=${yyyymm[i]} --hiveconf pre_year_mm=${prev_yyyymm[i]} -f my_hive_script.hql
done
    
mail -s "Script $0 completed." $EMAIL 
exit

Save the above example as my_hive_script.sh. Remember to use nohup when executing the script

nohup ./my_hive_script.sh

Bugs when switch from Window to Unix

Suppose you are using a Window system and want to execute your scripts on your Unix servers. You may have some bugs of mismatch decodes between the two systesms (it took me a few hours to figure out the bug.) To fix it, you can use Notepad++ to convert the format as follow:

Notepad++ > Edit > Unix (LF)

Then, save it and copy to your Unix server before running commands.

Bash Scripting for Hive was originally published by Long H. Nguyen at Long's homepage on August 13, 2018.

Hive Performance Checklist

2018-06-05T00:00:00+00:00

Apache Hive is a data warehouse software project built on top of Hadoop for providing data summarization, query and analysis. Hive is a SQL-like language so that it is easy to use and becomes popular to programmers. However, one of the biggest challenges Hive users face is the slow response time.

Below is my collection checklist for optimizing Hive performance:

USE TEZ (Hive 0.13+)

Apache Tez provides more efficient processing than the MapReduce execution engine, by reducing operations and limiting the amount of intermediate data that is written to disk.

As show in below figure, the traditional MapReduce execution engine has several steps in which the intermediate data from the reducers are written back to HDFS, which incurs the performance penalty for disk I/O. Contrast this with the data flow of the Tez execution engine shown on the right side, where the reducer’s intermediate data is passed directly to the next reducer in the execution plan and bypasses the expense of writing the data to disk.

set hive.execution.engine=tez;
set hive.prewarm.enabled=true;
set hive.prewarm.numcontainers=10;

USE Optimized Row Columnar (ORC) Format

The ORC format is a column-based storage format, meaning that rather than storing all of the data for an individual row of data consecutively on disk, the data for each column of storage contiguously instead. This helps to avoid unnecessary disk access for queries that do not contain certain columns, by “skipping over” large sections of data not needed in the results.

CREATE TABLE A_ORC (
customerID int, name string, age int, address string
) STORED AS ORC tblproperties (“orc.compress" = “SNAPPY”);

USE VECTORIZATION

Vectorized query execution improves performance of operations like scans, aggregations, filters and joins, by performing them in batches of 1024 rows at once instead of single row each time. Introduced in Hive 0.13, this feature significantly improves query execution time, and is easily enabled with two parameters settings:

set hive.vectorized.execution.enabled = true;
set hive.vectorized.execution.reduce.enabled = true;

COST BASED QUERY OPTIMIZATION (Hive0.14+)

The cost-based optimization (CBO) engine uses statistics in the Hive Metastore to produce optimal query plans. There are two types of statistics that are used for optimization: table stats, which include the uncompressed size of the table, number of rows, and number of files used to store the data, and column stats, which include NDV (number of distinct values) and min/max/count values.

The CBO does join reordering, improves plans for star and bushy join schemas, and provides opportunistic improvements based on sample queries. The downside of the CBO is the fact that you must gather and maintain accurate statistics about your tables in order for the cost-based optimization engine to be effective. Unfortunately, the collection of table statistics is an expensive operation, but the benefits can be reaped on all subsequent queries involving the table for which statistics were collected.

set hive.cbo.enable=true;
set hive.compute.query.using.stats=true;
set hive.stats.fetch.column.stats=true;
set hive.stats.fetch.partition.stats=true;
ANALYZE TABLE weather_ORC COMPUTE STATISTICS;

Use STREAMTABLE for JOIN queries

Joins play an important role and the order of joining table becomes very important. When doing table joins, Hadoop starts from the last table listed and joins to the left. Most of us put the largest table as the first table in the list, meaning it is last in the joining process. This then requires more resources to process.

Put the large table as the last table in the list so Hadoop starts here
Or, use SELECT /* +STREAMTABLE(<table name>) */ …. Streamtable tells Hadoop which table to start with.

Use MAPJOIN to speedup join queries

Mapjoin is a little-known feature of Hive. It allows a table to be loaded into memory so that a (very fast) join could be performed entirely within a mapper without having to use a Map/Reduce step. If your queries frequently rely on small table joins (e.g. cities or countries, etc.) you might see a very substantial speed-up from using mapjoins. For example:

SELECT /*+ MAPJOIN(c) */ * FROM orders o JOIN cities c ON (o.city_id = c.id);

Alternative way is to turn on mapjoins is to let Hive do it automatically. Simply set hive.auto.convert.join to true in your config, and Hive will automatically use mapjoins for any tables smaller than hive.mapjoin.smalltable.filesize (default is 25MB).

Use OLAP functionality (OVER and RANK) instead of Join wherever possible

For example, we would like to find the latest URL for each sessionID. One might consider the following approach:

SELECT click_event.* FROM click_event inner join 
(select session_ID, max(time_stamps) as max_ts from click_event group by session_ID) latest ON click_event.session_ID = latest.session_ID
AND click_event.time_stamps = latest.max_ts;

In the above query, we build a sub-query to collect the timestamp of the latest event in each session, and then use an inner join to filter out the rest. While the query is a reasonable solution—from a functional point of view—it turns out there is a better way to re-write this query as follows:

SELECT * FROM
(SELECT *, RANK() over (partition by session_ID, order by time_stamps DESC) as rank FROM click_event) ranked_clicks
WHERE ranked_clicks.rank=1;

Use SORT BY and DISTRIBUTE BY instead of ORDER BY wherever possible

ORDER BY takes only single reducer to process the data which may take an unacceptably long time to execute for longer data sets. Hive provides an alternative, SORT BY, that orders the data only within each reducer and performs a local ordering where each reducer’s output will be sorted. Better performance is traded for total ordering. In both cases, the syntax differs only by the use of the ORDER or SORT keyword. We can specify any columns you wish and specify whether or not the columns are ascending using the ASC keyword (the default) or descending using the DESC keyword. Here is an example using ORDER BY:

SELECT s.year_month_date, s.symbol, s.price_close
FROM stocks s
ORDER BY s.year_month_date ASC, s.symbol DESC;

Here is the same example using SORT BY instead:

SELECT s.year_month_date, s.symbol, s.price_close
FROM stocks s
SORT BY s.year_month_date ASC, s.symbol DESC;

The two queries look almost identical, but in the second case, if more than one reducer is invoked, the output will be sorted differently. While each reducer’s output files will be sorted, the data will probably overlap with the output of other reducers. Here Hive provides DISTRIBUTE BY with SORT BY which controls how map output is divided among reduces. The idea is like all data that flows through a MapReduce job is organized into key-value pairs and Hive must use this feature internally when it converts your queries to MapReduce jobs. By default, MapReduce computes a hash on the keys output by mappers and tries to evenly distribute the key-value pairs among the available reducers using the hash values.

SELECT s.year_month_date, s.symbol, s.price_close
FROM stocks s
DISTRIBUTE BY s.symbol
SORT BY s.symbol ASC, s.year_month_date ASC;

DISTRIBUTE BY works similar to GROUP BY in the sense that it controls how reducers receive rows for processing, while SORT BY controls the sorting of data inside the reducer.

HAVING clause for filtering the rows not for other purpose

HAVING clause is used to filter the rows after all the rows are selected. It is just like a filter. We should avoid HAVING clause for any other purposes. Considering below example:

Use:

SELECT cellName, count(cellName)
FROM internal_channel_switching
WHERE cellName != ‘1059’ AND cellName != ‘5730’
GROUP BY cellName;

Instead of:

SELECT cellName, count(cellName)
FROM internal_channel_switching
GROUP BY cellName
HAVING cellName != ‘1059’ AND cellName != ‘5730’;

Minimize number of subquery blocks in the query

Sometimes you may have more than one subqueries in your main query. Try to minimize the number of subquery block in your query. Considering below example:

Use:

SELECT name
FROM employee
WHERE (salary, age) = (SELECT MAX (salary), MAX (age)
FROM employee_details) AND emp_dept = ‘Computer Science;

Instead of:

SELECT name
FROM employee
WHERE salary = (SELECT MAX(salary) FROM employee_details)
AND age = (SELECT MAX(age) FROM employee_details) AND emp_dept = ‘Computer Science’;

Use operator EXISTS, IN and table joins appropriately in your query

Usually IN has the slowest performance.
IN is efficient when most of the filter criteria is in the sub-query.
EXISTS is efficient when most of the filter criteria is in the main query.

Considering below example

Use:

Select * from product p
where EXISTS (select * from order_items o
where o.product_id = p.product_id);

Instead of:

Select * from product p
where product_id IN (select product_id from order_items);

Give importance to the conditions in WHERE clause

Recommended

SELECT emp_id, first_name, salary FROM employee WHERE salary > 50000;

Not recommended

SELECT emp_id, first_name, salary FROM employee WHERE salary != 50000;

Recommended

SELECT emp_id, first_name, salary
FROM employee
WHERE first_name LIKE 'Pravat%';

Not recommended

SELECT emp_id, first_name, salary
FROM employee
WHERE SUBSTR(first_name,1,3) = 'Pra';

Recommended

SELECT emp_id, first_name, salary
FROM employee
WHERE first_name LIKE NVL ( :name, '%');

Not Recommended

SELECT emp_id, first_name, salary
FROM employee
WHERE first_name = NVL ( :name, first_name);

Recommended

SELECT product_id, product_name
FROM product
WHERE unit_price BETWEEN MAX(unit_price) and MIN(unit_price);

Not Recommended

SELECT product_id, product_name
FROM product
WHERE unit_price >= MAX(unit_price)
and unit_price <= MIN(unit_price);

Recommended

SELECT emp_id, first_name, salary
FROM employee WHERE dept = 'ComputerScience'
AND location = 'Singapore';

Not Recommended

SELECT emp_id, first_name, salary
FROM employee
WHERE dept || location= 'ComputerScienceSingapore';

Use non-column expression on one side of the query because it will be processed earlier.

Hive Performance Checklist was originally published by Long H. Nguyen at Long's homepage on June 05, 2018.

Introduction to R for data science

2018-04-30T00:00:00+00:00

My deck for introduction to R for data science. Below is its outline:

Why R is popular for data science?
Essentials of R Programming
Important packages for each steps in data mining.
- Importing data
- Data Manipulation
- Data Visualization
- Data modeling
- Reporting
Examples with R codes are included.

Introduction to R for data science was originally published by Long H. Nguyen at Long's homepage on April 30, 2018.

Metrics for model evaluation

2017-09-23T00:00:00+00:00

There are a few popular metrics for model evaluation:

Gain chart: = Gain at a given decile level is the ratio of cumulative number of targets (events) up to that decile to the total number of targets (events) in the entire data set.
Lift chart: Lift is the ratio of gain % to the random expectation % at a given decile level.
KS chart: K-S is a measure of the degree of separation between the positive and negative distributions. It is a very popular metrics in credit risk modeling.
AUC plot: The ROC curve is created by plotting the true positive rate (TPR) against the false positive rate (FPR) at various threshold settings. This plot visualizes overall perfomance of models and is very useful metric for imbalance data.

This post will provide R code for plotting the charts using ggplot2 and computing these metrics so that you can reuse it easily.

Library

library(ROCR)
library(ggplot2)
set.seed(1)
n=10000
# Randomly create score and ground truth for examples
score <- runif(n)
y <- (runif(n) < score)

Gain chart

gain.chart <- function(score, y) {
  ## Shuffle predicition
  set.seed(0)
  rand <- sample(1:length(y))
  score <- score[rand]
  y <- y[rand]
  
  pred <- prediction(score, y)
  perf <- performance(pred, measure = "tpr", x.measure = "rpp")
  plot.data <- data.frame(xvals=unlist(perf@x.values), yvals=unlist(perf@y.values))
  theme_update(plot.title = element_text(hjust = 0.5))
  print(ggplot(plot.data, aes(x=xvals, ymin=0, ymax=yvals)) +
    geom_ribbon(alpha=0.2) + geom_line(aes(y=yvals)) + scale_x_continuous(breaks = seq(0, 1, by = 0.2)) +
    ggtitle("Gain chart") + xlab('% of Population') + ylab('% of Responders'))
  results.sortedByProb<-order(score,decreasing=TRUE)
  test_y.sorted <-y[results.sortedByProb]
  for(k in 1:10){
    cat('Gain ratio at ',k*10,'% is: ',round(sum(100*test_y.sorted[1:(length(test_y.sorted)*k/10)])/sum(test_y.sorted),digits=2),'% \n')
  }
  for(k in 1:10){
    cat('# Responders at ',k*10,'% is: ',round(sum(test_y.sorted[(length(y)*(k-1)/10+1):(length(y)*k/10)])),'\n')    
  }
}

Lift chart

lift.chart <- function(score, y) {
  pred <- prediction(score, y)
  perf <- performance(pred, measure = "lift", x.measure = "rpp")
  plot(perf, main="Lift curve", col = 'green')
}

KS

KS <- function(score, y){
  perf <- performance(prediction(score, y), "tpr", "fpr")
  ks <- max(attr(perf, "y.values")[[1]] - (attr(perf, "x.values")[[1]]))
  return(ks)
}

AUC chart

roc.chart <- function(score, y) {
  pred <- prediction(score, y)
  perf <- performance(pred, measure = "tpr", x.measure = "fpr")
  auc <- performance(pred, measure = "auc")
  auc <- auc@y.values[[1]]*100
  plot.data <- data.frame(xvals=unlist(perf@x.values), yvals=unlist(perf@y.values))
  theme_update(plot.title = element_text(hjust = 0.5))
  print(ggplot(plot.data, aes(x=xvals, ymin=0, ymax=yvals)) +
    geom_ribbon(alpha=0.2) + geom_line(aes(y=yvals)) + scale_x_continuous(breaks = seq(0, 1, by = 0.2)) +
    ggtitle(paste0("ROC Curve with AUC=", round(auc,digits = 2),"%")) + xlab('FPR') + ylab('TPR'))
  return(auc)
}

Metrics for model evaluation was originally published by Long H. Nguyen at Long's homepage on September 23, 2017.

SAS snippet code

2017-03-16T00:00:00+00:00

Collection of small and re-usable source code for SAS

Feel free to use it if you find it helpful.

Count number of rows/observation in a table

/*define a macro function*/
%macro obsct (inpdat);
data _null_;
put nobs=;
stop;
set &inpdat nobs=nobs;
run;
%mend;
/*call the macro function*/
%obsct (Your_table);

Import data from .CSV file

data your_table;  
infile 'your_table.csv' delimiter = ','  firstobs=2;  
length col1 $16. col2 $20. col3 $20.; 
input   col1 $   col2 $   col3 $   col4   col5   col6 ; 
run;

Print data summary

PROC MEANS DATA=your_table N Q1 Median Q3 QRANGE Mean std Min Max MAXDEC=3;
	title 'Summary of your_table';
	VAR var1 var2 var3;
RUN;

SAS snippet code was originally published by Long H. Nguyen at Long's homepage on March 16, 2017.

R snippet code

2017-03-16T00:00:00+00:00

My collection of small and re-usable source code for R.

Feel free to use it if you find it helpful.

Correlation plot

It could be the first step of your data analytics project. You need to visually pair-wise distribution of your data and how the variables are correlated to each others.

corrgram(data,
         main="Correlation matrix for your data",
         lower.panel=panel.pts, upper.panel=panel.cor,
         diag.panel=panel.density)

Overlay data distributions for comparisons

Given two groups of data, we may need to compare their distributions. Overlay distribution plots will be helpful with beautiful visualization.

# assign two groups for legends
group1$partition = 'group1'
group2$partition = 'group2'
pltData = rbind(group1,group2)
# reset your data
group1$partition = NULL
group2$partition = NULL
for( var in imp_vars){
  xl = paste(var,'\n')
  xl = paste(xl, 'group1: ',round(mean(group1[[var]],na.rm = T),2), '+/-', round(sd(group1[[var]],na.rm = T),2),'\n')
  xl = paste(xl, 'group2: ',round(mean(group2[[var]],na.rm = T),2), '+/-', round(sd(group2[[var]],na.rm = T),2),'\n')
  print(ggplot(pltData, aes(pltData[[var]], fill = partition)) + geom_density(alpha = 0.2) + xlab(xl))
  cat(xl)
  invisible(readline(prompt="Press [enter] to continue"))
}

Histogram of important features vs label

One of my sample code taken from a Kaggle competition.

It is very helpful to create ‘golden’ features. :)

library(readr)
library(ggplot2)
library(ggthemes)
train <- read_csv("../input/train.csv")
#This important list is generated by my model, here I just provide top 10.
important_list = c('PropertyField37','SalesField5','PersonalField9','Field7'
                  ,'PersonalField2','PersonalField1','SalesField4','PersonalField10A'
                  , 'SalesField1B', 'PersonalField10B', 'PersonalField12')
train$QuoteConversion_Flag = as.factor(train$QuoteConversion_Flag)
for( att in important_list){
#Density histogram
plot <- ggplot(train, aes_string(att, fill = 'QuoteConversion_Flag')) +
  geom_histogram(alpha = 0.5, position = 'identity') +
  ggtitle(paste0('Histogram of attribute ', att))
}

For example, Field7 at bin value ~25 could be useful for creating an additional variable.

R snippet code was originally published by Long H. Nguyen at Long's homepage on March 16, 2017.

Set up Caffe to run Deep learning

2016-06-23T00:00:00+00:00

Caffe is a fast high performance Deep neural network library. It requires Nvidia GPU with CUDA support.

Install CUDA

Download CUDA for Linux from here: (https://developer.nvidia.com/cuda-downloads). For me, I download runfile (local)

Run sudo sh cuda_7.5.18_linux.run. Follow instructions.

Then, you may encounter this error: “You appear to be running an X server”. Follow this link to fix it: (http://askubuntu.com/questions/149206/how-to-install-nvidia-run)

Hit CTRL+ALT+F1 and login using your credentials.
Kill your current X server session by typing sudo service lightdm stop or sudo stop lightdm
Enter runlevel 3 by typing sudo init 3 and install your *.run file.
You might be required to reboot when the installation finishes. If not, run sudo service lightdm start or sudo start lightdm to start your X server again.

Install cuDNN

cuDNN is used by caffe for GPU acceleration, provided by CUDA, very much recommended for speed. Go to (https://developer.nvidia.com/cuDNN), download .tar.gz file and extract. There are 2 things to do with this.

Copy all the files, (except cudnn.h) to /usr/local/cuda-6.5/lib64
Copy the cudnn.h to /usr/local/cuda-6.5/include

Again, you may have an error here: “ImportError: libcudart.so.7.0: cannot open shared object file: No such file or directory”.

Run the following command:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/lib64
sudo ldconfig /usr/local/cuda/lib64

Install Caffe

Clone Caffe from GitHub (https://github.com/BVLC/caffe) and follow the installation instructions (http://caffe.berkeleyvision.org/installation.html)
Install all the dependencies and libraries
- sudo apt-get install libprotobuf-dev libleveldb-dev libsnappy-dev libopencv-dev libboost-all-dev libhdf5-serial-dev
- sudo apt-get install libgflags-dev libgoogle-glog-dev liblmdb-dev protobuf-compiler
Edit config file
- USE_CUDNN := 1
Compile caffe
- make all
- make test
- make runtest
Build also python module and distribution libraries.
- make pycaffe
- make distribute
Check out important examples

Set up Caffe to run Deep learning was originally published by Long H. Nguyen at Long's homepage on June 23, 2016.

How to join Kaggle competitions for learning and fun

2016-03-26T00:00:00+00:00

Kaggle is probably a well-known place for data science competitions with more than a half of million users. It bridges the gap between data scientists who, need data to play with and companies, which require top quality solutions for theirs business problems.

Sample code of my Kaggle competitions.

In my opinion, it is good to join Kaggle competitions for learning and fun. Firstly, you surely can learn a lot from these competitions. Each competition typically has more than 2000 participants; therefore, you definitely learn from this crowd. You can learn from the forum, scripts, and especially from the winning solutions. Secondly, it is fun since this is a place, where you can make new friends. You may form a team for the competitions and meet up for discussions.

The next question is how a person can effectively join Kaggle competitions? Here, I’d like to share some tips from my experience so that you can save your time and learn new knowledge in an effective way. Below are my basic steps when I join a competition.

Structure your folders: It is good to practice a standard folder structure when colloberating with your partners. You will need to organize our folders and don’t want to spend so much time just for changing folder paths.
- .\input: Folder for input data, feature data
- .\R: Folder for R source code
- .\python: Folder for python code
- .\submission: Folder for submissions
- .\stacking: Folder for stacking
- .\model: Folder for saving trained models
- .\figure: Folder for saving plots, figures
Moreover, you may need to use GitLab for versioning and reproducing your code.
Understand the dataset: After creating your folder structure, it is the time to download the competition’s data. It is worthy to find answers of the following questions:
- Does the dataset contain missing values? Should missing values be replace by NA, median, -1, or -999? Typically, for a non-negative and numerical attribute/column, replacing missing values by -999 will work well.
- How many ordinal columns are there in the dataset? Ordinal columns can be replaced by numbers. However, please consider magnitude gaps between ordinal values.
- How many category/string column are there in the dataset? How many unique values does each category column have?
- For each numerical column, what are its statistics, e.g, min, 1st quartile, median, mean, 3rd quartile, max. Histograms of each column should be plotted for further investigation.
- What are the competition’s problem? Classification or regression? Evaluation metric? Is the dataset imbalance?
Check the forum: Once you understand the dataset and competition’s problem, it is time to read the forum for a quick catch-up on what participants have done.
- Note down information on data quality discussions, which models work well.
- Find good starting scripts to save your data pre-processing time.
Test a variety of models: You may have a set of machine learning models (which you have used for previous competitions) and want to test whether they can work well for this dataset. Below are some models for testing perfomance. Please take note that simple model will contribute more for later stacking step.
- Logistic regession
- Ridge, Lasso, ElasticNet, and SGD classifiers
- Support Vector Machine (SVM)
- Random forest (RF)
- Extra Tree Classifier (ETC)
- Gradient Boosting Machine (GBM)
- Xgboost (XGB)
Fine-tune the best model After testing a variety of models, it is a good time to fine-tune the best model for satisfying the temptation of getting a higher rank. :D
- One-by-one parameter tunnining vs. grid search: There are both GridSearch (python) and caret (R) for hyper-parameter tunning. However, these built-in functions are very expensive in terms of computing resource. My approach is tunning each parameter one-by-one. I usually vary one parameter and keep the rest unchange for finding the best value of the changing paramter. The process is repeated for all parameters.
- Naming convention for your models and submission: It is important to name your models and submission in a consitent way so that you can easily compare different models later. My naming structure for submissions is as follow:
  
  [test | train]_(CV score)_(my name)_(model name)_features_(number of features)_(parameter i)_(value i)
For example, I name a submission generated by a XGB model having a CV score of 0.967, 450 features, a depth value of 5, etc.

test_0.967_Long_xgb_features450_depth5_eta0.023_round2486_sub0.9_col0.77.csv

CV prediction of this model on the training dataset be saved similar with “train” prefix and the model is without prefix in the folder “./model/”
Feature selection: Next, we need to remove redundent features and select the core set of features. It helps to reduce the complexity of the model while retaining comperative perfomance based on the full feature set. Moreover, it also helps to significantly reduce the numbers of new features that you will create in the next step. It is very convenient that RF, ETC, GBM, and XGB have built-in feature selection method, which is based on coverage of attribures. You can call the important_feature() function easily.
Feature engineering: People say feature engineering is the winning key. It is essenstial to check important articles about feature engineering in my reading list link.

To be updated
Stacking: To be updated

How to join Kaggle competitions for learning and fun was originally published by Long H. Nguyen at Long's homepage on March 26, 2016.

Reading list for data science

2016-01-10T00:00:00+00:00

Today, I visit my LinkedIn and would like to retrieve my posts, which I shared in the past. Unfortunately, LinkedIn only maintainces my posts in the last two weeks. It is really disappointed. Hence, I would like to mantaince my reading list for data science by myself.

The readling list will be sorted into following categories:

Data Science News
Data Science Background
Data Manupulating
Data Visualization
Feature Engineering
Data Modelling
Kaggle
Interview Questions

Link for my list

Reading list for data science was originally published by Long H. Nguyen at Long's homepage on January 10, 2016.

Start my blog with Jekyll

2015-12-16T00:00:00+00:00

Basic steps to build your own blog with Jekyll

Host your homepage with GitHub
Set up your custom domain
Using Jekyll with Pages
- If you use Window, please refer this site
Select your favourite theme, fork and modify it.
- For me, I selected Minimal Mistake theme
Blog like a hacker Jekyll & Markdown
- bundle exec jekyll serve

Start my blog with Jekyll was originally published by Long H. Nguyen at Long's homepage on December 16, 2015.