Architecture/Technology advice for pipeline using Hadoop/Hive

Vote count:

0

My architecture is built of couple of stages.

1. ETL putting files on HDFS file system.
2. Hive running sql scripts on top of Hadoop and generating result set table.
3. The table is converted into XML
4. the XML is being uploaded to another location using http post.

We found our self having logic on Hive sql's and bash scripts. not sure if that's the right way of doing this.

I am looking for a pipleline framework to help me nail this architecture(Java/Spring or any other).

Any suggestions? examples? I tried PIG but we have complications with that.

Thanks, ray.

asked 1 min ago

rayman

2,021

Architecture/Technology advice for pipeline using Hadoop/Hive