Tricky | Tricky problem? Well, stop looking, you've just found the solution!

DataStage: 64bit version of DataStage with 64bit Oracle Database on Windows

If you are trying to connect DataStage 64bit on Windows to Oracle Database 64bit you probably get error message like this:

DataStage was unable to attach to this connector. Error loading connector library ccora11g.dll. 
%1 is not a valid Win32 application.

1 2	DataStage was unable to attach to this connector. Error loading connector library ccora11g.dll. %1 is not a valid Win32 application.

I assume that you have done all prerequisites like set up the environment variables etc. More about prerequisites for Oracle connector in DataStage V11.3 you can find here http://ibm.co/1mu4WGo This error message you… Read More »

Datastage: APT_Communicator::connectTo: connect() failed due to Unix error = 10,061

If you are getting the error message while running the job:

APT_Communicator::connectTo: connect() failed due to Unix error = 10,061 (Unknown error) on node <HOSTNAME> using ConnectionInfo object 'TCP,  connection Host: HOSTNAME (<IP>), TCP port number: 0', RETRYING connect()

1	APT_Communicator::connectTo: connect() failed due to Unix error = 10,061 (Unknown error) on node <HOSTNAME> using ConnectionInfo object 'TCP, connection Host: HOSTNAME (<IP>), TCP port number: 0', RETRYING connect()

You have to follow these steps to fix the issue: You must ensure that you have in hosts file these two entries

127.0.0.1 localhost
::1 localhost

1 2	127.0.0.1 localhost ::1 localhost

Next step is try if this commands working fine

ping -4 localhost
ping -6 localhost

1 2	ping -4 localhost ping -6 localhost

Set the parameter $APT_USE_IPV4 in DataStage project to True (1) Open… Read More »

Uncompressing files into HDFS (Wikipedia dump example)

In case you want to test your Big Data environment, there is one very simple way, how to do it: Download Wikipedia dump and analyze it with Word Count application. We will focus on the first step – downloading the data and loading it into HDFS. This can be achieved by several ways. Today I’ll… Read More »

How to convert hexadecimal number to IP address in C++

When you need to convert hexadecimal number in C++ like 0DF52286 to standard IP address format like 13.245.34.134 you can use this code:

static char* hex_to_ip(const char *input)
{
    char *output = (char*)malloc(sizeof(char) * 16);
    unsigned int a, b, c, d;

    if (sscanf(input, "%2x%2x%2x%2x", &a, &b, &c, &d) != 4)
        return output;
    sprintf(output, "%u.%u.%u.%u", a, b, c, d);
    return output;
}

static char* hex_to_ip(const char *input)

{

char *output = (char*)malloc(sizeof(char) * 16);

unsigned int a, b, c, d;

if (sscanf(input, "%2x%2x%2x%2x", &a, &b, &c, &d) != 4)

return output;

sprintf(output, "%u.%u.%u.%u", a, b, c, d);

return output;

}

The method takes one argument which is the IP address in hexadecimal format.

Datastage error: Field “NAME” has keyprep export buffer overrun error

While running a job, you getting following error:

Field "NAME" has keyprep export buffer overrun error

1	Field "NAME" has keyprep export buffer overrun error

This error happens during processing data in Lookup stage and the solution of this error is very simple. Check the key attributes in the lookup stage and ensure that all attributes have same type and length (check the extended parameter too).

How to generate random number in DataStage

Very simple procedure to generate random number in DataStage. Grab a transformer on your canvas and use following code:

Mod(Rand(),MAXNUMBER)

1	Mod(Rand(),MAXNUMBER)

This code will return you a pseudorandom number in the range from zero to (MAXNUMBER – 1). So for example Mod(Rand(),500) will return number from 0 to 499.

How to cleanup old Datasets in DataStage automatically

This is a procedure useful not only for DataStage environment. Often you need to delete old logs periodically in order to save disk space. But especially in DataStage, which lacks automatic datasets cleanup is handy to delete old datasets. These may originate from your DataStage Dataset stages, or as a temporary resource from Lookup stage… Read More »