Get Twitter Trends Using Spring Batch

Spring Batch

                      In this article we will see a sample Spring Batch application. We will try to use Twitter Trends API to get the trends response and parse it. The below diagram depicts the system architecture of the Twitter Trends Extraction application.

Twitter Trends Extractor

 

                  Now we will see the ItemReader implementation. The ItemReader makes API call to Twitter to get the trends. Here we have used Twitter4j to authenticate and to make API calls to twitter.

TwitterReader Class Diagram


package org.smarttechie.twitter.reader;

import org.springframework.batch.item.ItemReader;
import org.springframework.batch.item.NonTransientResourceException;
import org.springframework.batch.item.ParseException;
import org.springframework.batch.item.UnexpectedInputException;
import org.springframework.stereotype.Component;

import twitter4j.Trends;
import twitter4j.Twitter;
import twitter4j.TwitterFactory;
import twitter4j.auth.AccessToken;

/**
* The TwitterTrendsReader implements ItemReader. This will make API call to twitter and pass the trends to processor.
* @author Siva Janapati
* @version 1.0
*/

@Component("reader")
public class TwitterTrendsReader implements ItemReader<Trends> {
/**
 * This methods make API call to twitter and pass the trends to processor
 */
 @Override
 public Trends read() throws Exception, UnexpectedInputException,
 ParseException, NonTransientResourceException {
    TwitterFactory factory = new TwitterFactory();
    AccessToken accessToken = new AccessToken("<Access Token Here>", "<Access Token Secret Here>");
    Twitter twitter = factory.getInstance();
    twitter.setOAuthConsumer("<Consumer Key (API Key)>", "<Consumer Secret (API Secret)>");
    twitter.setOAuthAccessToken(accessToken);
    Trends trends = twitter.getPlaceTrends(1);
    return trends;
   }
}

Now we will see the ItemProcessor implementation. The ItemProcessor converts the twitter trend object to our own domain model.

TwitterProcessor Class Diagram


package org.smarttechie.twitter.processor;

import java.util.ArrayList;
import java.util.List;

import org.smarttechie.twitter.model.TwitterTrend;
import org.springframework.batch.item.ItemProcessor;
import org.springframework.stereotype.Component;

import twitter4j.Trends;

/**
* The TwitterTrendsProcessor implements ItemProcessor. This will convert the Trends object to our own domain model.
* @author Siva Janapati
* @version 1.0
*/
@Component("processor")
public class TwitterTrendsProcessor implements ItemProcessor<Trends, List<TwitterTrend>> {
/**
* This method converts the Trends object to our own domain model.
*/
@Override
public List<TwitterTrend> process(Trends trends) throws Exception {
    List<TwitterTrend> twitterTrends = null;
    TwitterTrend twitterTrend = null;
    if (trends != null && trends.getTrends() != null && trends.getTrends().length > 0) {
       twitterTrends = new ArrayList<TwitterTrend>();
         for (int index = 0; index < trends.getTrends().length; index++) {
             twitterTrend = new TwitterTrend();
             twitterTrend.setName(trends.getTrends()[index].getName());
             twitterTrend.setQuery(trends.getTrends()[index].getQuery());
             twitterTrend.setUrl(trends.getTrends()[index].getURL());
             twitterTrends.add(twitterTrend);
      }
    }
    return twitterTrends;
  }
}

              Now we will see the ItemWriter implementation. The ItemWriter can write the data to any data source (flat file, database etc…). Spring Batch has OOTB implementations for FlatFileItemWriter, JdbcBatchItemWriter etc… You can find the list of OOTB ItemReders and ItemsWrites here.

TwitterWriter Class Diagram


package org.smarttechie.twitter.writer;

import java.util.List;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.smarttechie.twitter.model.TwitterTrend;
import org.springframework.batch.item.ItemWriter;
import org.springframework.stereotype.Component;
/**
* TwitterTrendsWriter {@link ItemWriter} which logs the trends information to console.
* @author Siva Janapati
* @version 1.0
*/
@Component("writer")
public class TwitterTrendsWriter implements ItemWriter<List<TwitterTrend>> {

private static final Log log = LogFactory.getLog(TwitterTrendsWriter.class);
@Override
public void write(List<? extends List<TwitterTrend>> items)
throws Exception {
    if (items != null && !items.isEmpty()) {
       List<TwitterTrend> trends = items.get(0);
         if (trends != null && !trends.isEmpty()) {
            for (TwitterTrend trend : trends) {
               log.info("The name : " + trend.getName());
               log.info("The query : " + trend.getQuery());
               log.info("The url : " + trend.getUrl());
       }
     }
   }
 }
}

The sequence diagram depicts the interaction of the above components.

Twitter Trends Spring Batch

The Spring Batch configuration is given below.

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:batch="http://www.springframework.org/schema/batch"
xmlns:context="http://www.springframework.org/schema/context"
xmlns:jdbc="http://www.springframework.org/schema/jdbc"
xsi:schemaLocation="
http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch-2.1.xsd
http://www.springframework.org/schema/jdbc http://www.springframework.org/schema/jdbc/spring-jdbc-3.0.xsd
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.0.xsd">

 <context:component-scan base-package="org.smarttechie.twitter" />

 <bean id="jobRepository" class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean">
     <property name="transactionManager" ref="transactionManager" />
 </bean>

 <bean id="transactionManager" class="org.springframework.batch.support.transaction.ResourcelessTransactionManager" />

 <batch:job id="job1">
    <batch:step id="step1">
        <batch:tasklet transaction-manager="transactionManager">
           <batch:chunk reader="reader" writer="writer" processor="processor" commit-interval="1" />
        </batch:tasklet>
    </batch:step>
 </batch:job>
</beans>

Launch the application using CommandLineJobRunner by providing the batch configuration and the job id. The code created for this article is available here.

That’s it. The Spring Batch sample application is ready.

Happy Batch Programming !!!

Advertisements

I am Siva Prasad Rao Janapati. Working as a software developer. Has hands on experience on ATG Commerce(DAS/DPS/DCS), Mozu commerce, Broadleaf Commerce, Java, JEE, Spring, Play, JPA, Hibernate, Velocity, JMS, Jboss, Weblogic,Tomcat, Jetty, Apache, Apache Solr, Spring Batch, JQuery, NodeJS, SOAP, REST, MySQL, Oracle, Mongo DB, Memcached, HazelCast, Git, SVN, CVS, Ant, Maven, Gradle, Amazon Web services, Rackspace, Quartz, JMeter, Junit, Open NLP, Facebook Graph,Twitter4J, YouTube Gdata, Bazzarvoice,Yotpo, 4-Tell, Alatest, Shopzilla, Linkshare. I have hands on experience on open sources and commercial technologies.

Tagged with: ,
Posted in Spring, Spring Batch

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

DZone

DZone MVB

Java Code Geeks
Java Code Geeks
%d bloggers like this: