Knowledge is Power
brew packages Basic Apps $ brew install bash $ brew install bash-completion $ brew install maven $ brew install openssl $ brew install ssh-copy-id $ brew install wget $ brew install gawk Big data Apps $ brew install hadoop $...
Pre-requisites User account for which passwordless ssh will be setup, should be present on all nodes Password of the account should be same across all nodes pdsh and ssh-copy-id commands should be available Prepare 2 files file_of_hosts.txt – containing all...
Lately I tried installing xz/lzma codec on my local VM setup. The compression ratios are pretty awesome. Won’t do a benchmark here, try it out yourself 😉 Steps Download codec JAR – https://github.com/yongtang/hadoop-xz or https://mvnrepository.com/artifact/io.sensesecure/hadoop-xz Copy downloaded JAR to HDPs’ libs...
So you done everything right and can’t connect Excel to Hive using ODBC driver on your macOS? Let’s see what is going on. Are you running El Capitan on Sierra? Well I was running Sierra and tried connecting before while...
There are times when we want to parse a string that is actually a JSON. Usually that could be done with built in functions of Hive such as get_json_object(). Though get_json_object cannot parse JSON Array from my experience. These array...
At times we want access to tmux and we are just stuck due to admin restrictions or root access. There are three important parts to setting up tmux, It is dependent on libevent Compiling and installing on non-system folders require customized...
Something that is very much desired while working on shell. I love to have this feature of $ssh ser[TAB] Using brew is my preferred way, just type in – $brew install bash-completion Do not forget to add the text in...
Source: https://community.hortonworks.com/articles/16846/how-to-identify-what-is-consuming-space-in-hdfs.html Script #!/usr/bin/env bash max_depth=5 largest_root_dirs=$(hdfs dfs -du -s ‘/*’ | sort -nr | perl -ane ‘print “$F[1] “‘) printf “%15s %s\n” “bytes” “directory” for ld in $largest_root_dirs; do printf “%15.0f %s\n” $(hdfs dfs -du -s $ld| cut -d’ ‘...
Source: https://azure.microsoft.com/en-us/documentation/articles/hdinsight-linux-ambari-ssh-tunnel/ Original Author: Larry Franks Excerpts ssh tunnel command ssh -C2qTnNf -D 9876 user-name@machine-name This creates a connection that routes traffic to local port 9876 to the cluster over SSH. The options are: D 9876 – The local port that will route...
Source: https://community.hortonworks.com/articles/16846/how-to-identify-what-is-consuming-space-in-hdfs.html Find the directories using the most space in HDFS For a UI showing the biggest consumers of space in HDFS install and configure Twitter’s HDFS-DU. For a quick visual representation of HDFS disk usage with no extra tools required,...
More
