How JustAnswer Works:

  • Ask an Expert
    Experts are full of valuable knowledge and are ready to help with any question. Credentials confirmed by a Fortune 500 verification firm.
  • Get a Professional Answer
    Via email, text message, or notification as you wait on our site.
    Ask follow up questions if you need to.
  • 100% Satisfaction Guarantee
    Rate the answer you receive.

Ask Steve Herrod Your Own Question

Steve Herrod
Steve Herrod, Computer Support Specialist
Category: Software
Satisfied Customers: 160
Experience:  Familiar with a wide variety of software and experienced in user training/support
65126503
Type Your Software Question Here...
Steve Herrod is online now

Hadoop contains some parallelization technology and a

Customer Question

Hello, Hadoop contains some parallelization technology and a protocol of communication between the master and the slave nodes. Is it any different from MPI or it is only a kind of copy?
Submitted: 5 months ago.
Category: Software
Expert:  JamesI replied 5 months ago.

Hadoop supports fault tolerance (node failure, corrupt data transports etc), which is not the case of MPI.

Most implementations I have seen of Hadoop have also used MPI, since the two technologies specialise in different tasks, and it comes down to the calculations and data manipulation you are performing.

If you have resilient and high performance hardware, MPI can be used, however where there is a risk of failure\latency Hadoop helps reduce some of the effects.

That said Hadoop is aimed at non-iterative algorithms where nodes require little data exchange to proceed (non-iterative and independent), where as MPI is aimed at iterative algorithms where nodes require data exchange to proceed (iterative and dependent).

I found a good explanation of the two frameworks (which explain it better than me)

  1. MPI is a message passing paradigm of parallelism. Here, you have a root machine which spawns programs on all the machines in its MPI WORLD. All the threads in the system are independent and hence the only way of communication between them is through messages over network. The network bandwidth and throughput is one of the most crucial factor in MPI implementation's performance. Idea : If there is just one thread per machine and you have many cores on it, you can use OpenMP shared memory paradigm for solving subsets of your problem on one machine.
  2. Hadoop is used for solving large problems on commodity hardware using Map Reduce paradigm. Hence, you do not have to worry about distributing data or managing corner cases. Hadoop also provides a file system HDFS for storing data on compute nodes.
Customer: replied 4 months ago.
Thanks your informative answer. Please clarify and I will rate you : "The parallelization algorithms of MPI are not integrated in Hadoop MapReduce and thus Hadoop parallelization is something original".

What Customers are Saying:

 
 
 
  • My Expert answered my question promptly and he resolved the issue totally. This is a great service. I am so glad I found it I will definitely use the service again if needed. One Happy Customer
< Previous | Next >
  • My Expert answered my question promptly and he resolved the issue totally. This is a great service. I am so glad I found it I will definitely use the service again if needed. One Happy Customer
  • Wonderful service, prompt, efficient, and accurate. Couldn't have asked for more. I cannot thank you enough for your help. Mary C.
  • This expert is wonderful. They truly know what they are talking about, and they actually care about you. They really helped put my nerves at ease. Thank you so much!!!! Alex
  • Thank you for all your help. It is nice to know that this service is here for people like myself, who need answers fast and are not sure who to consult. GP
  • I couldn't be more satisfied! This is the site I will always come to when I need a second opinion. Justin
  • Just let me say that this encounter has been entirely professional and most helpful. I liked that I could ask additional questions and get answered in a very short turn around. Esther
  • Wonderful service, prompt, efficient, and accurate. Couldn't have asked for more. I cannot thank you enough for your help. Mary C.
 
 
 

Meet The Experts:

 
 
 
  • Steve Herrod

    Steve Herrod

    Computer Support Specialist

    Satisfied Customers:

    160
    Familiar with a wide variety of software and experienced in user training/support
< Last | Next >
  • http://ww2.justanswer.com/uploads/SP/spherrod/2012-6-6_174244_1000852.64x64.JPG Steve Herrod's Avatar

    Steve Herrod

    Computer Support Specialist

    Satisfied Customers:

    160
    Familiar with a wide variety of software and experienced in user training/support
  • http://ww2.justanswer.com/uploads/SE/Seashore2011/2011-12-6_33418_111205223015.64x64.jpg Chris L.'s Avatar

    Chris L.

    Support Specialist

    Satisfied Customers:

    114
    Certified Software expert with over 10 years experience.
  • http://ww2.justanswer.com/uploads/JE/jessmagz/2012-6-6_18129_jm.64x64.jpg Jess M.'s Avatar

    Jess M.

    Computer Support Specialist

    Satisfied Customers:

    112
    Computer Software Support specialist for more that 10 years
  • http://ww2.justanswer.com/uploads/FS/fszcze/2012-6-18_181848_500test.64x64.jpg Frederick S's Avatar

    Frederick S

    Computer Specialist

    Satisfied Customers:

    91
    Computer technician; founder of a home PC repair company.
  • http://ww2.justanswer.com/uploads/WA/wa94sound/2012-6-6_202633_iStock000007010121Small.64x64.jpg Bernie's Avatar

    Bernie

    Software Support

    Satisfied Customers:

    52
    6+ years experience, Microsoft MCDST/CompTia A+ Certified
  • http://ww2.justanswer.com/uploads/CH/chutz747/2011-11-6_14231_me.64x64.jpg The-PC-Guy's Avatar

    The-PC-Guy

    Software Engineer

    Satisfied Customers:

    38
    20 years experience providing remote computer support
  • http://ww2.justanswer.com/uploads/jnayes/2010-12-23_132454_japicbetter.jpg jnayes's Avatar

    jnayes

    Computer Support

    Satisfied Customers:

    37
    Network Administrator