Tricky | Tricky problem? Well, stop looking, you've just found the solution!

HDFS Node Manager: FATAL Error in dispatcher thread

When your HDFS Node Manager does not start for no apparent reason (such as following one), solution is simple: check free space on your disk. Node Manager gets nervous when there is less than 10 % of free disk space and may not work properly or at all.

2016-03-17 17:23:16,426 FATAL event.AsyncDispatcher (AsyncDispatcher.java:dispatch(181)) - Error in dispatcher thread
java.lang.NullPointerException
       at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:345)
       at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
       at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:449)
       at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$PublicLocalizer.addResource(ResourceLocalizationService.java:797)
       at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker.handle(ResourceLocalizationService.java:704)
       at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker.handle(ResourceLocalizationService.java:646)
       at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:175)
       at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:108)
       at java.lang.Thread.run(Thread.java:745)

2016-03-17 17:23:16,426 FATAL event.AsyncDispatcher (AsyncDispatcher.java:dispatch(181)) - Error in dispatcher thread

java.lang.NullPointerException

at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:345)

at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)

at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:449)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$PublicLocalizer.addResource(ResourceLocalizationService.java:797)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker.handle(ResourceLocalizationService.java:704)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker.handle(ResourceLocalizationService.java:646)

at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:175)

at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:108)

at java.lang.Thread.run(Thread.java:745)

Source: my very own experience :)

Ambari service “Start” including “Start all” are greyed out

You might run into mentioned situation after Hadoop cluster or node reboot. Solution is quite simple: Ambari agent is apparently not running on each node or at all. Start it with following command:

ambari-agent start

1	ambari-agent start

In case your are unable to access your Ambari web interface at all (you wouldn’t be able to see previous error, but… Read More »

Služební cesty: nedobrovolným poutníkem na kraj světa a občas i zpět

Definice Služebka, business trip, nebo prostě jakýkoliv jiný výlet, kdy za cizí peníze obětujete vlastní zdraví i rozum. Pěkně korporátně, kde jsou jasná pravidla a procesy, které se ale stejně jasně dají otáčet, ohýbat, nedejbože i obejít. Čistě subjektivní obrázek, neeticky generalizující, genderově i politicky nekorektní. Tedy skoro přesně tak jak se mi to vydralo… Read More »

[Salesforce.com] Regex too complicated

Working with the regular expressions in Salesforce.com is same as in the Java. But the regex have restrictions to its complexity. For example, this regex throws compilation error:

([a-zA-Z]*_*)*\.([a-zA-Z]*_*)*\.[0-9]*

1	([a-zA-Z]_)\.([a-zA-Z]_)\.[0-9]*

If you have to use multiple times the symbol * in your regex, the Salesforce.com evaluates your pattern as too complicated. Workaround The workaround is to… Read More »

[Salesforce.com] Upsert on the User object

If you are trying to do the UPSERT operation on the User object, for example:

User theUser = new User();
...
theUser.FederationIdentifier = 'FederationIdentifier';

upsert theUser FederationIdentifier;

User theUser = new User();

...

theUser.FederationIdentifier = 'FederationIdentifier';

upsert theUser FederationIdentifier;

you will get error:

DML not allowed on User

1	DML not allowed on User

Workaround The workaround for this case is use the Database object:

User theUser = new User();
...
theUser.FederationIdentifier = 'FederationIdentifier';

Database.upsert(theUser, User.Fields.FederationIdentifier, true);

User theUser = new User();

...

theUser.FederationIdentifier = 'FederationIdentifier';

Database.upsert(theUser, User.Fields.FederationIdentifier, true);

…and now you can happily use the upsert operation on the User object :)

Docker can’t connect to boot2docker (OSX)

When you are trying to connect docker to boot2docker on OSX, you probably got this error message:

fata 0032 an error occurred trying to connect i/o timeout

1	fata 0032 an error occurred trying to connect i/o timeout

Solution to fix this error: Go to the Virtual Box -> Preferences -> Network -> Host-only networks Delete the host only networks and create one new. Then open the command line run this commands: Warning: The boot2docker… Read More »

Enterprise Data Hackathon 2014 očima účastníka

03. 03. 2014 Při čtení padákova blogu o Data Hackathonu v NODE5 přemýšlím, že na podobné akci by se dalo přiučit spoustě věcí. Obzvláště když věci jako K-Clustering nebo RF analýza mi absolutně nic neříkají. Nechávám to ale volnému průběhu a doufám, že třeba příští rok…

Encoding error when importing DataStage project

Common error when importing project from machine to another may be with encoding:

File XYZ.dsx can only be imported when the client code page is 1252 - it is currently 1250

1	File XYZ.dsx can only be imported when the client code page is 1252 - it is currently 1250

You can solve it very easily – just open the file you want to import and change CharacterSet attribute to desired encoding:

CharacterSet="CP1250"

1	CharacterSet="CP1250"

Simple, huh?

DataStage Oracle connector error: The variable bInvalidDateTime has value 1

If you encounter an error The variable bInvalidDateTime has value 1 while running your job containing output Oracle stage, it is most likely caused by non-date value inserted to date column in target database. In case you want to ignore this error and insert a NULL instead of non-valid date (or timestamp), there is a… Read More »

IBM Cognos: XQE-JDB-0004 A problem occurred finding the driver class “com.ibm.db2.jcc.DB2Driver”.

When you are trying to test the connection to the DB2 database from Cognos configuration and you are getting this error message: XQE-JDB-0004 A problem occurred finding the driver class “com.ibm.db2.jcc.DB2Driver”. This error cause because the Cognos cannot find the DB2 drivers. Solution of this problem is simple. You have to copy files db2jcc.jar and… Read More »