Spring YARN and Hortonworks Sandbox

Spring YARN and Hortonworks Sandbox

after much trial and error, here's some notes on getting Spring YARN to submit a remote job to Hortonworks sandbox...

1. if working "remotely" (launching YARN job from one machine into the sandbox) make sure your local /etc/hosts is set to resolve sandbox.hortonworks.com on your target machine IP
2. the ResourceManager port on HDP is 8050, not the default 8032 (spring.hadoop.resourceManagerPort: 8050)
3. when running the job, use "HADOOP_USER_NAME=hdfs java -jar [your jar file]

Spring XD JMS Source using Spring Integration

Spring XD - JMS input

Spring XD is a fantastic new framework to help ingest data from a variety of inputs and process them out.  there's a bunch of 'source' points that come with Spring XD, but one that i wanted to use was JMS (grab my JMS messages and stick it in Hadoop…)  so, here's a simple one that works with Spring XD 1.0.0.M1

i'm using Spring Integration, so here's the JMS listener

here's the connection factory for the database

now this is what i need to do to add it to my Spring XD;

• add the jms-source-module.xml to $SPRING_XD/modules/source
• add the package jar to $SPRING_XD/lib
• add the following 'supporting' jars
• activemq-core-5.5.1.jar
• spring-integration-jms-2.2.3.RELEASE.jar
• spring-jms-3.2.1.RELEASE.jar
• geronimo-jms_1.1_spec-1.1.1.jar
• geronimo-j2ee-management_1.1_spec-1.0.1.jar

now start up the activemq server

$ACTIVE_MQ/bin/activemq start

and startup spring-xd

$SPRING_XD/bin/xd-singlenode

now to start it the 'listener'

curl -d "jms-source-module | log" http://localhost:8080/streams/jmstest

or if you're going to use hadoop as the consumer

curl -d "jms-source-module | hdfs --rollover=10" http://localhost:8080/streams/jmstest

and send a message (via the console is an easy way to test)

now, you should see it show up in the log file of spring xd (remember, "log" was specified after the pipe)

to see it in hadoop, try the following

./hadoop fs -ls /xd/jmstest

Setup a Single-Node Hadoop Cluster on Ubuntu

Setup a Single-Node Hadoop Cluster on Ubuntu

a nice step-by-step to setup Hadoop on Ubuntu

http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/

only thing done differently is the manual install of Java 6 (that ppa doesn't appear to work)

https://help.ubuntu.com/community/Java