Apche Spark (behind the scenes) | Coursera Community

Apche Spark (behind the scenes)

  • 24 March 2021
  • 4 replies
  • 58 views

Userlevel 2
Badge +2

This post is linked to my post on a coursera forum, because adding images does not currently work on coursera forums Interface, I had to post the rest of my idea posted on a forum here (this is an idea that I support by illustrations and that I couldn’t add on the forum interface of coursera), so in the forum I refer to these  images by adding the link  of this post. it is about : the need of understanding the "behind the secnes" of Apache Spark to really understand "The magic that happens with this framework .."

code & behind the scenes (with Spark master Web UI)

Note: you can click on the pictures to enlarge them

 


4 replies

Userlevel 4
Badge +7

Cool. I know Ambari also shows these DAG visualizations for different jobs.

Userlevel 4
Badge +7

These visualizations are useful because you can identify redundant operations and actually speed up your program. Apache spark already does this but lets say you were using pure Map/Reduce. You could see where you are not optimal and speed up using Tez.

 

 

Userlevel 4
Badge +7

You can also use Tez with Hive or Pig

Userlevel 4
Badge +7

You can go further an even create and visualize a sequence of lets say Spark jobs, Hive jobs, pure Map/Reduce jobs, ‘X’,’Y’,’Z’. Apache oozie allows you to schedule, visualize and execute these different jobs as you need. Cool, isn’t it?

 

 

Reply