Hadoop | Tricky

HDFS Node Manager: FATAL Error in dispatcher thread

When your HDFS Node Manager does not start for no apparent reason (such as following one), solution is simple: check free space on your disk. Node Manager gets nervous when there is less than 10 % of free disk space and may not work properly or at all.

2016-03-17 17:23:16,426 FATAL event.AsyncDispatcher (AsyncDispatcher.java:dispatch(181)) - Error in dispatcher thread
java.lang.NullPointerException
       at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:345)
       at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
       at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:449)
       at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$PublicLocalizer.addResource(ResourceLocalizationService.java:797)
       at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker.handle(ResourceLocalizationService.java:704)
       at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker.handle(ResourceLocalizationService.java:646)
       at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:175)
       at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:108)
       at java.lang.Thread.run(Thread.java:745)

2016-03-17 17:23:16,426 FATAL event.AsyncDispatcher (AsyncDispatcher.java:dispatch(181)) - Error in dispatcher thread

java.lang.NullPointerException

at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:345)

at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)

at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:449)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$PublicLocalizer.addResource(ResourceLocalizationService.java:797)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker.handle(ResourceLocalizationService.java:704)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker.handle(ResourceLocalizationService.java:646)

at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:175)

at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:108)

at java.lang.Thread.run(Thread.java:745)

Source: my very own experience :)

Ambari service “Start” including “Start all” are greyed out

You might run into mentioned situation after Hadoop cluster or node reboot. Solution is quite simple: Ambari agent is apparently not running on each node or at all. Start it with following command:

ambari-agent start

1	ambari-agent start

In case your are unable to access your Ambari web interface at all (you wouldn’t be able to see previous error, but… Read More »