spark installation in ubuntu 18.04

 UBUNTU COMMANDS TO INSTALL SBT echo "deb https://dl.bintray.com/sbt/debian /" | sudo tee -a /etc/apt/sources.list.d/sbt.list sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 2EE0EA64E40A89B84B2DF73499E82A75642AC823 sudo apt-get update sudo apt-get install sbt # the following command will take several minutes. it will finally start a sbt shell. sbt # exit the shell by typing exit at the prompt, i.e. exit -------------------- In a separate terminal the steps below can be executed. --------- cd /usr/lib/spark/conf sudo cp log4j.properties.template log4j.properties sudo vi log4j.properties change INFO to ERROR as below: log4j.rootCategory=INFO, console to log4j.rootCategory=ERROR, console exit log4j.properties after saving it. -- not needed---------- sudo apt-get remove --purge openjdk* java-common default-jdk sudo apt-get remove --purge sbt sudo apt-get autoremove --purge # Check weather you successfully removed all related packages by: sudo update-alternat...

1. Download Spark: http://spark.apache.org/downloads.html (Steps 1, 2, 3, 4) 2. Unzip and move spark to /usr/lib/ machineName:~$tar xzvf spark-2.3.1-bin-hadoop2.7.tgz machineName:~$mv spark-2.3.1-bin-hadoop2.7 spark machineName:~$sudo mv spark /usr/lib/ 3. Install Java 8 following steps given in the web page below: http://tipsonubuntu.com/2016/07/31/install-oracle-java-8-9-ubuntu-16-04-linux-mint-18/ sudo add-apt-repository ppa:webupd8team/java sudo apt update; sudo apt install oracle-java8-installer Click OK and YES when the Oracle User Agreement screens appear. javac -version sudo apt install oracle-java8-set-default 4. Add Path: sudo nano ~/.bashrc 5. At the end of .bashrc, add the following and save: export JAVA_HOME=/usr/lib/jvm/java-8-oracle export SPARK_HOME=/usr/lib/spark/ export PATH=$PATH:$SPARK_HOME/bin 6. Open a new terminal and verify installation by running the spark shell: machineName:~ spark-shell