Are you ready to see GraphQL in action?

GraphQL                      In the last article, we have discussed GraphQL advantages over REST. In this article, we will see GraphQL in action. I have created a sample application to showcase differences between REST and GraphQL. First, we will see the REST implementation of simple product detail endpoint. I have used Spring Boot to demonstrate REST. Download sample project and follow the steps outlined in README to set up the project. I am not discussing setup details here as it is out of scope for this article. Assuming that your project is up and running to make a call to http://localhost:8080/product/{product_id} endpoint to get product detail JSON as shown below.

rest.gif

If you observe above JSON, we are getting entire product JSON including reviews and technical specifications though we are not interested in all the elements of a given product.

                   Now we will see GraphQL in action by getting product details in a selective manner. To demonstrate GraphQL again I used Spring Boot. Download sample project and follow the steps outlined in README to set up the project. I am not discussing setup details here as it is out of scope for this article. Assuming that your project is up and running to see GraphQL in action. In this case, I am interested to get only product id, title, short description and list price of a given product. Let us see how we can query to get interesting details.

grapgql.gif

Now as a service consumer I am interested to get product id, title, short description, list price, and reviews. In this case, GraphQL gives the flexibility to query what we want. See below query and response when we use GraphQL.

graphql2.gif

To demonstrate GraphQL I have used GUI based plugin GraphiQL. For consuming from other applications we can configure endpoint in application.properties.

graphql.servlet.mapping=/graphql
graphql.servlet.enabled=true
graphql.servlet.corsEnabled=true

Now we can make a call to the above endpoint by passing  URL encoded query parameter as shown below. You can learn more about query and mutations https://graphql.org/learn/queries/

GraphQL_Query

Hope you enjoyed this article. I will come back with another article. Till then, Happy Learning!!!

Tagged with: , , ,
Posted in GraphQL, Spring Boot

Are you ready to adopt GraphQL?

 

GraphQL                             In this article let us explore GraphQL. Let us first understand what GraphQL is? GraphQL is a specification from Facebook. GraphQL is a query language for APIs and runtime for fulfilling those queries with your existing data. GraphQL gives clients the power to ask for exactly what they need and nothing more by avoiding over fetching or under fetching of data. We can understand it more when we are going to see GraphQL implementation in action. Till then hold your curiosity. 

Wait, Wait… So far we are using REST(Representation State Transfer) to expose our services as APIs. Let us ask some questions ourselves before getting deeper into GraphQL.

Why do I need to adopt GraphQL?
What problems am I facing with REST APIs? How GraphQL solves those?
                     To answer the above queries let us take a use case to build an e-commerce application for web, mobile and native clients. We decided to expose APIs for various e-commerce functionalities.  For example, I have product detail REST API which gives specific product information as JSON which includes product data attributes, specifications data, reviews data, etc. As we are having many attributes in product JSON, the size of it is more.  Each client (web and thin clients(mobile and tablets))  has it’s own front end requirements to display product data as they have different screen sizes, memory, network bandwidth, etc. Now my clients started consuming product detail API. Though mobile and tablet interfaces don’t require entire product JSON as web, still product detail API is giving entire product data. It is evident that clients don’t have control over the data what they want from the server. This is called over fetching. The pictorial representation of REST over fetching issue is given below.REST                        We can solve over fetching issue with various approaches. The straight forward approach is to maintain different APIs for thick and thin clients. Though this design solves over fetching issue but has other problems like code maintenance,  implementation of enhancements across different APIs, deployment of thick, thin client APIs, more compute, more manpower, etc., which puts more cost to project. The other approach is having middleware to intercept the client request. Based on client request filter the response to return. This adds an additional layer to the application which has the same issues as the previous approach.

                   Now let us discuss the second issue with REST called under fetching. To avoid over fetching, we decided to create granular APIs so that clients will make API calls for whichever the data they required. Let us take a product detail page for the web. It has product information, specifications and reviews to display. Now to render product detail page client is not going to get data in a single API call. So client needs to make multiple API calls (like basic product API, specification API, Reviews API) to cater to its data requirement. This design has performance issues with an increased number of round trips to the backend server and APIGateway. The other issue is requiring more computing power and network as rising in the number of requests to serve. Below is a pictorial representation of under fetching.REST Under Fetching                 Let us see the third issue with REST that is, evolving APIs with versions. Any API will evolve as business needs will change with time. As per our customer needs, we might need to add data attributes(most of the cases we won’t remove data attributes as we need to have backward compatibility) to existing APIs. When we do any changes to existing APIs, we need extra vigilant as the changes might break the clients. To avoid that we will do versioning of APIs as and when we plan to release changes to existing APIs. When we introduce new versions which put the burden of managing more APIs(i.e. more compute power, more manpower), planning to deprecate older versions. Discipline and communication are needed when we have more versions of an API. With REST we cannot do silent releases.

                  The above issues are leading us to look for another solution called GraphQL. We will see how GraphQL addresses the above-said issues by implementing an API in the upcoming article. Meanwhile, let us see the request and response paradigm with GraphQL and how GraphQL makes clients happy by serving what they want.GraphQL Here are some of the adopters of GraphQL https://graphql.org/users/.
In the coming article, we will see the implementation of an API with GraphQL. Till then Spread love for APIs!!!

 

 

Tagged with: , , ,
Posted in GraphQL

APIGEE: CI/CD Pipeline for API Proxies

Apigee

                                       In this article, we will see how to create a CI/CD pipeline for APIGEE API proxies. I have referred a couple of articles on APIGEE community on the same topic. Those gave some idea on how to setup CI/CD pipeline for API proxies. Here are the tools which I have used to setup CI/CD.

  1. Jenkins
  2. NodeJs
  3. apigeelint
  4. newman
  5. APIGEE Management APIs

Make sure that you have created APIGEE edge account and a sample proxy to start with.  Below is the architecture diagram which shows the CI/CD pipeline and the stages involved. You can use this as a baseline CI/CD for your projects and can enhance it based on your requirements.

APIGEE_CI_CD

Here are the steps I have implemented in CI/CD pipeline.

  1. Developer pushes the API proxy code to GIT.
  2. Jenkins polls GIT and starts CI/CD Stage 1 based on GIT changes.
  3. As part of Stage 1, the code will be pulled into the workspace.
  4. In “Static Code Analysis” stage, the code will be analyzed for any violations of best code practices and anti-patterns usage. If this stage is the success it proceeds with the build stage. After each stage completion either success or failure, the notification will be sent to Slack channel.
  5. As part of the build stage, we will create APIGEE API proxy bundle.
  6. In the Deploy stage, I used APIGEE management APIS to deploy the API proxy bundle.
  7. Once the deployment is successful, then the integration tests will be triggered. I used Newman to do integration tests. Newman requires integration tests collection file as input. The test cases can be created easily with Postman
  8. In all the stages the notifications will be triggered to Slack channel.

There are some enhancements which I will do in the coming days. Below are some of the changes which I will target as enhancements.

  • Adding email, Hipchat notifications
  • Revert the API proxy to a previous revision if the integration tests fail.
  • If integration tests success, promote build to load test environment and run load test scripts.

The setup and project used as part of this article are available on GitHub. Till then, Spread love for APIs!!!

Apigee Message Logging Policy Demo

 

Tagged with:
Posted in APIGEE

APIGEE API gateway log management with ELK(Elastic Search, Logstash and Kibana)

Apigee

                  In this article, we will see how we can manage logs generated by APIGEE API gateway with Elastic Search, Logstash and Kibana. The below diagram highlights how log data flows through ELK stack for data visualization and monitoring.

apigee_log_management

As part of the API gateway, we will use MessageLogging policy to capture API proxy logs at proxy flow, target flow and post client flow based on the requirement. As of today, MessageLogging policy supports Syslog and file-based logging(available for only on-premise installations). Below is MessageLogging policy configuration to send logs to the syslog server.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<MessageLogging async="false" continueOnError="false" enabled="true" name="TestEnv-Message-Logging">
<DisplayName>TestEnv Message Logging</DisplayName>
<Syslog>
<Message>{messageId}|{apiproxy.name}|{response.status.code}|{target.received.start.timestamp}|{target.received.end.timestamp}|{environment.name}|{client.received.start.timestamp}|{client.sent.end.timestamp}</Message>
<Host>127.0.0.1</Host>
<Port>514</Port>
</Syslog>
</MessageLogging>

Now we will see logstash configuration to ingest data from syslog to logstash. Here I am not giving steps to setup ELK as that is out of scope.

input {
file {
path => "/Users/sjana/Documents/POC/ELK/APIGEE_LOG.log"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
csv {
separator => "|"
columns => ["MessageId","API_PROXY_NAME","RESPONSE_STATUS_CODE","TARGET_RECEIVED_START_TIME","TARGET_RECEIVED_END_TIME","ENVIRONMENT_NAME","CLIENT_RECEIVED_START_TIME","CLEINT_SENT_END_TIME"]
remove_field => "message"
}
}
output {
elasticsearch {
hosts => "http://localhost:9200&quot;
index => "apiproxies-log"
}
stdout {}
}

Once Syslog data is ingested successfully to Logstash we can configure visualization based on available fields in the index. Below is sample visualization created for API proxy response status codes.

kibana_visualization

In coming articles we will discuss another topic. Till then, Spread love for APIs!!!

Tagged with: ,
Posted in APIGEE

APIGEE – An Introduction to API Gateway

Apigee

In this article I want to give brief introduction about APIGEE. Mainly APIGEE offers below functionalities out of the box as api gateway.

  1. Protocol Transformation
    Transform from or to any protocol including SOAP, REST, XML binary, or custom
  2. Traffic Management
    Flexible, distributed quota management, rate limiting, and spike arrest policies out-of-the-box
  3. API Security
    Built-in support for address filtering, JSON and XML schema validation, and bot detection
  4. Data Access & Security
    Two-way SSL/TLS, API key validation, OAuth1, OAuth2, SAML, CORS, encrypted store, and HIPAA and PCI compliance
  5. API Products
    Create different tiers by packaging APIs with varying rate limits and pricing
  6. API Analytics
    Fine-grained performance monitoring, including anomaly tracing and drill-down and usage metrics related to apps, developers, and APIs
  7. API Monetization
    Flexible rate plans, international billing, and usage tracking
  8. Global Policy Management
    Enforce consistent security and governance policies across all APIs
  9. Developer Portal
    A customizable portal for API providers to manage developers, APIs, and API documentation and versioning

APIGEE sits between service consumers and backend services. As an API gateway, APIGEE takes care of common functionalities required for the APIs. Hence backend services can concentrate only on the core business logic. The below digram  depicts where exactly APIGEE fits.

APIGEE

Now we will see how the request and response goes through APIGEE. Every client request will go through proxy and target endpoints where we will attach APIGEE policies. Policy is nothing but a simple module which will provide common API functionality which we can configure using XML. In each flow APIGEE exposes lot of flow variables.

APIGEE (2).png

Below are list of policies available out of the box which you can attach to proxy and target flows.

APIGEE (3)

If you want to experience APIGEE edge, follow below steps to create trail account.

  1. Goto https://login.apigee.com/sign_upEnter the required information to create account with APIGEE.
  2. After activating APIGEE account, login to https://login.apigee.com/login access Apigee Edge management UI

Below are some of the reference materials to deep dive into APIGEE

 

Tagged with:
Posted in APIGEE

APIGEE – Can we configure parameters for Message Logging Policy?

Apigee

Apigee MessageLogging policy has limitation to use configurable parameters such as Syslog server host, port, etc. details. If we are not going to configure these parameters we may get into trouble while moving the proxy from one environment to another environment. To achieve the portability the approach is to have MessageLogging policy for each environment. Based on the environment in which the proxy is running the policy will be applied. Below is the sample proxy with message logging policy for each environment. The proxy definition is given below.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ProxyEndpoint name="default">
<Description/>
<FaultRules/>
<PreFlow name="PreFlow">
<Request/>
<Response/>
</PreFlow>
<PostFlow name="PostFlow">
<Request/>
<Response>
<Step>
<Name>TestEnv-Message-Logging</Name>
<Condition>environment.name = "test"</Condition>
</Step>
<Step>
<Name>ProdEnv-Message-Logging</Name>
<Condition>environment.name = "prod"</Condition>
</Step>
</Response>
</PostFlow>
<Flows/>
<HTTPProxyConnection>
<BasePath>/messgageloggingdemo</BasePath>
<Properties/>
<VirtualHost>default</VirtualHost>
<VirtualHost>secure</VirtualHost>
</HTTPProxyConnection>
<RouteRule name="noroute"/>
</ProxyEndpoint>

The test and prod environment MessageLogging policy configuration is given below.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<MessageLogging async="false" continueOnError="false" enabled="true" name="TestEnv-Message-Logging">
<DisplayName>TestEnv Message Logging</DisplayName>
<Syslog>
<Message>{environment.name}</Message>
<Host>10.0.0.1</Host>
<Port>556</Port>
</Syslog>
</MessageLogging>

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<MessageLogging async="false" continueOnError="false" enabled="true" name="ProdEnv-Message-Logging">
<DisplayName>ProdEnv Message Logging</DisplayName>
<Syslog>
<Message>{environment.name}</Message>
<Host>10.0.0.2</Host>
<Port>448</Port>
</Syslog>
</MessageLogging>

The proxy demonstrated is available on GitHub to download and play with it.

Apigee Message Logging Policy Demo

Tagged with:
Posted in APIGEE

APIGEE – How To Handle Base64 Encoding Decoding?

Apigee

In this article, we will see how to encode and decode base64 strings while building APIGEE proxies. As part of APIGEE, we have BasicAuthentication policy which deals with base64 encoded authorization header. But if we want to deal with any base64 encoded string other than Authorization header we should go with JavaScript policy or JavaCallout policy or PythonScript policy custom implementation. In this article, I will show you how to achieve base64 encode and decode using JavaScript policy.

Let me create a simple proxy with JavaScript policy to decode base64 encoded string. Below is the JavaScript policy configuration.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Javascript async="false" continueOnError="false" enabled="true" timeLimit="200" name="JS-Base64EncodeDecode">
<DisplayName>JS-Base64EncodeDecode</DisplayName>
<IncludeURL>jsc://Base64EncodeDecode.js</IncludeURL>
<ResourceURL>jsc://DecodeBase64String.js</ResourceURL>
</Javascript>

In the JavaScript policy, I have included the Base64EncodeDecode js file which performs the encode and decode. Below is the JavaScript to decode base64 encoded string.

var base64EncodedKey = context.getVariable("request.queryparam.key");
var key = Base64.decode(base64EncodedKey);
print(key);

The JavaScript which does base 64 encode and decode is available here.

The sample proxy created to demonstrate base64 encode and decode is available on GitHub. Download the sample proxy bundle and import to APIGEE Edge to play with it.

APIGEE

In next article we will discuss another use case. Till then Happy Coding!!!

 

Tagged with: ,
Posted in APIGEE

Java 11 Features – Java Flight Recorder

Java 11

                            In this article we will see how we can leverage Java Flight Recorder feature as part of Java 11. Earlier it was one of the commercial feature. But with Java 11 with JEP 328 this is open sourced to OpenJDK from OracleJDK. The Java Flight Recorder records the OS and JVM events to a file which can be inspected using Java Mission Control (JMC). Enabling JFR puts minimal overhead on the JVM performance. Hence this can be enabled for production deployments too. Now we will see some of the JVM arguments to enable JFR.

  • Time Based

java -XX:StartFlightRecording=delay=20s,duration=60s,filename=C:\myRecording.jfr,settings=profile,name=SampleRecording

view raw
gistfile1.txt
hosted with ❤ by GitHub

  • Continuous with dump on demand
java -XX:StartFlightRecording=settings=default

  • Continuous with dump on exit

java -XX:StartFlightRecording=settings=default -XX:FlightRecorderOptions=dumponexit=true,dumponexitpath=C:\tmp

As the JFR is built in available with Java 11, this excites the developer community. We can reduce the dependency on 3rd party profilers as well.

As part of Java 11 we are getting jdk.jfr module. This API allows programmers to produce custom JFR events and consume the JFR events stored in a file to troubleshoot the issue.

You can download the Java11 early access from http://jdk.java.net/11/ to explore the features.

Tagged with: ,
Posted in Java

Java 10 – Local Variable Type Inference

Java10
In this article we will see Java10 feature called Local Variable Type Inference proposed as part of JEP 286. From the first version of Java it is strongly typed language where we need to mention each variable data type. We all were feeling Java is verbose language and expecting precise, compact way of writing Java code. Java 8 addressed this concern some what. Java 10 added Local Variable Type Inference with initializer to eliminate verbosity. For example,

jshell> Map<String,String> map = new HashMap<>();
jshell> var map = new HashMap<>(); //This is valid with Java10

Here LHS variable datatype will be determined by RHS statement. For example,

jshell> var i = 3;
i ==> 3 //based on RHS, the LHS datatype is int.
jshell>int i=3,j=4; //Valid Declaration
but,
jshell> var j=4,k=5; //Not a Valid Declaration
| Error:
|'var' is not allowed in a compound declaration
| var j=4,k=5;
|^

You can use this feature for enhanced for loop and for loop as well.

jshell> List names = Arrays.asList("ABC","123","XYZ");
names ==> [ABC, 123, XYZ]
jshell> for(var name : names){
...> System.out.println("Name = "+ name);
...> }

Name = ABC
Name = 123
Name = XYZ

We can use Local Variable Type Inference in the for loop as well.


jshell> int[] arr = {1,2,3,4};
arr ==> int[4] { 1, 2, 3, 4 }

jshell> for (var i=0;i<arr.length;i++){
   ...> System.out.println("Value = "+i);
   ...> }
Value = 0
Value = 1
Value = 2
Value = 3

There are certain scenarios where this feature is not valid to use. For example,

  • Not valid for constructor variables
  • Not valid for instance variables
  • Not valid for method parameters
  • Not valid to assign NULL value
  • Not valid as return type

Let us see examples for above statements.


jshell> public class Sample {
   ...>    private var name = "xyz";
   ...>    public Sample(var name) {
   ...>     this.name=name;
   ...>    }
   ...>    public void printName(var name){
   ...>      System.out.println(name);
   ...>    }
   ...>    public var add(int a, int b) {
   ...>     return a+b;
   ...>    }
   ...> }
|  Error:
|  'var' is not allowed here
|     private var name = "xyz"; //Instance variable
|             ^-^
|  Error:
|  'var' is not allowed here
|     public Sample(var name) { //Constructor variable
|                   ^-^
|  Error:
|  'var' is not allowed here
|     public void printName(var name){ //Method parameter
|                           ^-^
|  Error:
|  'var' is not allowed here
|     public var add(int a, int b) { //Method return type
|            ^-^


jshell> public class Sample {
   ...>    
   ...>    public static void main(String[] args) {
   ...>     var s = null;
   ...>    }
   ...> }
|  Error:
|  cannot infer type for local variable s
|    (variable initializer is 'null')
|      var s = null;
|      ^-----------^

When we migrate the code from lower versions to Java10, we no need to worry about the Local Variable Type Inference as this has the backward compatibility.

In the coming post we will learn another topic. Till then stay tuned!

Posted in Java

Introduction to Apache Kafka

Apahe Kafka

What is Apache Kafka?

Apache Kafka is a distributed streaming system with publish and subscribe the stream of records. In another aspect, it is an enterprise messaging system. It is highly fast, horizontally scalable and fault tolerant system. Kafka has four core APIs called,

Producer API: 

This API allows the clients to connect to Kafka servers running in the cluster and publish the stream of records to one or more Kafka topics.

Consumer API:

This API allows the clients to connect to Kafka servers running in the cluster and consume the streams of records from one or more Kafka topics. Kafka consumers pull the messages from Kafka topics.

Streams API:

This API allows the clients to act as stream processors by consuming streams from one or more topics and producing the streams to other output topics. This allows to transform the input and output streams.

Connector API:

This API allows to write reusable producer and consumer code. For example, if we want to read data from any RDBMS to publish the data to the topic and consume data from the topic and write that to RDBMS. With connector API we can create reusable source and sink connector components for various data sources.

What use cases Kafka used for?

Kafka is used for the below use cases,

Messaging System:

Kafka used as an enterprise messaging system to decouple the source and target systems to exchange the data. Kafka provides high throughput with partitions and fault tolerance with replication compared to JMS.


Apache Kafka Messaging System

Web Activity Tracking:

To track the user journey events on the website for analytics and offline data processing.

Log Aggregation:

To process the log from various systems. Especially in the distributed environments, with microservices architectures where the systems are deployed on various hosts. We need to aggregate the logs from various systems and make the logs available in a central place for analysis. Go through the article on distributed logging architecture where Kafka is used https://smarttechie.org/2017/07/31/distributed-logging-architecture-for-micro-services/

Metrics Collector:

Kafka is used to collect the metrics from various systems and networks for operations monitoring. There are Kafka metrics reporters available for monitoring tools like Ganglia, Graphite, etc.

Some references on this https://github.com/stealthly/metrics-kafka

What is a broker?

An instance in a Kafka cluster is called a broker. In a Kafka cluster if you connect to anyone broker you will be able to access the entire cluster. The broker instance which we connect to access cluster is also known as the bootstrap server. Each broker is identified by a numeric id in the cluster. To start with Kafka cluster three brokers is a good number. But there are clusters which have hundreds of brokers in it.

What is a Topic?

A topic is a logical name to which the records are published. Internally the topic is divided into partitions to which the data is published. These partitions are distributed across the brokers in a cluster. For example, if a topic has three partitions with 3 brokers in the cluster each broker has one partition. The published data to partition is appended only with the offset increment.

Topic Partitions

Below are the couple of points we need to remember while working with partitions.

  • Topics are identified by its name. We can have many topics in a cluster.
  • The order of the messages is maintained at the partition level, not across the topic.
  • Once the data written to partition is not overridden. This is called the immutability.
  • The message in partitions is stored with key, value, and timestamp. Kafka ensures to publish the message to the same partition for a given key.
  • From the Kafka cluster, each partition will have a leader which will take read/write operations to that partition.

Apache Kafka Partitions

In the above example, I have created a topic with three partitions with replication factor 3. In this case, as the cluster is having 3 brokers, the three partitions are evenly distributed and the replicas of each partition are replicated over to another 2 brokers. As the replication factor is 3, there is no data loss even 2 brokers goes down. Always keep replication factor is greater than 1 and less than or equal to the number of brokers in the cluster. You can not create a topic with replication factor more than the number of brokers in a cluster.

In the above diagram, for each partition, there is a leader(glowing partition) and other in-sync replicas(gray out partitions) are followers. For partition 0, the broker-1 is leader and broker-2, broker-3 are followers. All the reads/writes to partition 0 will go to broker-1 and the same will be copied to broker-2 and broker-3.

Now let us create a Kafka cluster with 3 brokers by following the below steps.

Step 1:

Download the Apache Kafka latest version. In this example I am using 1.0 which is latest. Extract the folder and move into the bin folder. Start the Zookeeper which is essential to start with Kafka cluster. Zookeeper is the coordination service to manage the brokers, leader election for partitions and alerting the Kafka during the changes to topic ( delete topic, create topic etc…) or brokers( add broker, broker dies etc …). In this example I have started only one Zookeeper instance. In production environments we should have more Zookeeper instances to manage fail-over. With out Zookeeper Kafka cluster cannot work.

./zookeeper-server-start.sh ../config/zookeeper.properties

view raw
Start Zookeeper
hosted with ❤ by GitHub

Step 2:

Now start Kafka brokers. In this example we are going to start three brokers. Goto the config folder under Kafka root and copy the server.properties file 3 times and name it as server_1.properties, server_2.properties and server_3.properties. Change the below properties in those files.

#####server_1.properties#####
broker.id=1
listeners=PLAINTEXT://:9091
log.dirs=/tmp/kafka-logs-1
#####server_2.properties######
broker.id=2
listeners=PLAINTEXT://:9092
log.dirs=/tmp/kafka-logs-2
######server_3.properties#####
broker.id=3
listeners=PLAINTEXT://:9093
log.dirs=/tmp/kafka-logs-3

view raw
Kafka Broker Config
hosted with ❤ by GitHub

Now run the 3 brokers with the below commands.

###Start Broker 1 #######
./kafka-server-start.sh ../config/server_1.properties
###Start Broker 2 #######
./kafka-server-start.sh ../config/server_2.properties
###Start Broker 3 #######
./kafka-server-start.sh ../config/server_3.properties

Step 3:

Create a topic with below command.

./kafka-topics.sh –create –zookeeper localhost:2181 –replication-factor 3 –partitions 3 –topic first_topic

view raw
Kafka Topic Creation
hosted with ❤ by GitHub

Step 4:

Produce some messages to the topic created in above step by using Kafka console producer. For console producer mention any one of the broker address. That will be the bootstrap server to gain access to the entire cluster.

./kafka-console-producer.sh –broker-list localhost:9091 –topic first_topic
>First message
>Second message
>Third message
>Fourth message
>

Step 5:

Consume the messages using Kafka console consumer. For Kafka consumer mention any one of the broker address as bootstrap server. Remember while reading the messages you may not see the order. As the order is maintained at the partition level, not at the topic level.

./kafka-console-consumer.sh –bootstrap-server localhost:9092 –topic first_topic –from-beginning

If you want you can describe the topic to see how partitions are distributed and the the leader’s of each partition using below command.

./kafka-topics.sh –describe –zookeeper localhost:2181 –topic first_topic
#### The Result for the above command#####
Topic:first_topic PartitionCount:3 ReplicationFactor:3 Configs:
Topic: first_topic Partition: 0 Leader: 1 Replicas: 1,2,3 Isr: 1,2,3
Topic: first_topic Partition: 1 Leader: 2 Replicas: 2,3,1 Isr: 2,3,1
Topic: first_topic Partition: 2 Leader: 3 Replicas: 3,1,2 Isr: 3,1,2

view raw
Kafka Topic Describe
hosted with ❤ by GitHub

In the above description, broker-1 is the leader for partition:0 and broker-1, broker-2 and broker-3 has replicas of each partition.

In the next article we will see producer and consumer JAVA API. Till then, Happy Messaging!!!

Tagged with:
Posted in Apache Kafka
Dzone.com
DZone

DZone MVB

Java Code Geeks
Java Code Geeks
OpenSourceForYou