What Exactly Is Apache Camel

What exactly is Apache Camel?

My take to describe this in a more accessible way...

In order to understand what Apache Camel is, you need to understand what are Enterprise Integration Patterns.

Let's start with what we presumably already know: The Singleton pattern, the Factory pattern, etc; They are merely ways of organizing your solution to the problem, but they are not solutions themselves. These patterns were analyzed and extracted for the rest of us by the Gang of Four, when they published their book: Design Patterns. They saved some of us tremendous effort in thinking of how to best structure our code.

Much like the Gang of Four, Gregor Hohpe and Bobby Woolf authored the book Enterprise Integration Patterns (EIP) in which they propose and document a set of new patterns and blueprints for how we could best design large component-based systems, where components can be running on the same process or in a different machine.

They basically propose that we structure our system to be message oriented -- where components communicate with each others using messages as inputs and outputs and absolutely nothing else. They show us a complete set of patterns that we may choose from and implement in our different components that will together form the whole system.

So what is Apache Camel?

Apache Camel offers you the interfaces for the EIPs, the base objects, commonly needed implementations, debugging tools, a configuration system, and many other helpers which will save you a ton of time when you want to implement your solution to follow the EIPs.

Take MVC. MVC is pretty simple in theory and we could implement it without any framework help. But good MVC frameworks provide us with the structure ready-to-use and have gone the extra mile and thought out all the other "side" things you need when you create a large MVC project and that's why we use them most of the time.

That's exactly what Apache Camel is for EIPs. It's a complete production-ready framework for people who want to implement their solution to follow the EIPs.

Apache Camel File-Watch Without Spring

Apache Camel is a standalone integration framework to easily integrate different systems using well know and established EIP (Enterprise Integration Patterns).

It is not tied to Spring in any way at its core and has different deployment / integration models out of which is Spring / Spring Boot for the sake of easing its adoption and configuration for Spring framework users.

Out of the runtime contexts, you can use for example Camel Main component to run your application having a File Watch component setup to watch the /tmp/ directory change events:

The main application would look like the following:

public class FileWatchApplication {

    private FileWatchApplication() {
    }

    public static void main(String[] args) throws Exception {
        // use Camels Main class
        Main main = new Main(FileWatchApplication.class);
        // now keep the application running until the JVM is terminated (ctrl + c or sigterm)
        main.run(args);
    }
}

A simple RouteBuilder setting up your File Watch component would look like the following (note that it must be within the same (or child) package of the FileWatchApplication class):

public class FileWatchRouteBuilder extends RouteBuilder {

    @Override
    public void configure() throws Exception {
        // snippet configuration from the official documentation
        from("file-watch:///tmp/")
                .log("File event: ${header.CamelFileEventType} occurred on file ${header.CamelFileName} at ${header.CamelFileLastModified}");
    }
}

Apache Camel: What marches messages along?

Under the hood I believe camel is constructing a pure graph where each node is a Camel endpoint/processor, and each edge is a route between two endpoints (a source and a destination). This graph is precisely what RouteBuilder is building when you invoke its API. When you go to start() a Camel route, the graph is most likely validated and translated into a series of Runnables that need to be executed, and probably uses some kind of custom Executor or thread management to handle these Runnables.

Thus, the execution of the Runnables (processors processing messages as they arrive) are handled by this custom Executor. This is the mechanism that "marches messages along", although the order in which the tasks are queued up is driven by the overarching structure of the graph composed by RouteBuilder.

What is the CamelContext in apache?

After seeing the videos in YouTube, I knew about the Apache camel framework,
and from that I got the answer that camelcontext is nothing but context of the 'apache camel framework' framework.

As many framework have context like Spring have an applicationcontext, Ninja have a context, same the 'Apache camel framework' have a context and that is called 'camelcontext'.

So, it is the run-time system of Apache Camel (framework) and it connects its different concepts such as routes, components or endpoints.

Reference :
Basic Apache Camel Tutorial

How to install Apache Camel?

Camel, Java-projects and build-tools

Apache Camel works like any other Java library or framework, meaning that in order to use the framework you'll have to include its java binaries (i.e .jar files) to your project as dependencies. Now to make things simpler most developers use either Maven or Gradle to create, manage and build their java projects.

From these two I would recommend Maven as it seems to be the preferred option for Camel developers with most examples (including ones in official camel site) using it. To install maven follow the official how to install maven guide.

Maven archetypes

To create example camel project you can use maven archetypes which are basically project templates that can be used to create various types of new projects. If you're reading Camel in Action you might be better off using Camel version 2.x.x in your projects with Java Development Kit version 8. Camel 3.x.x is pretty similar so it should be fairly easy to learn that after learning the basics with 2.x.x.

After installing maven you can use your Java IDE (i.e IntelliJ, VSCode, Eclipse or Netbeans) to create project from maven archetype with groupId: org.apache.camel.archetypes artifactId: camel-archetype-java and version: 2.25.4

Or use maven command line command:

# Create new camel java project for Camel version 2.25.4
mvn archetype:generate -DarchetypeGroupId="org.apache.camel.archetypes" -DarchetypeArtifactId=camel-archetype-java -DarchetypeVersion="2.25.4"

Camel project

The new project should contain project file pom.xml where you can specify all the dependencies for your project. The the camel-archetype-java should have the following dependencies listed in the dependencies section of pom.xml.

<dependencies>

    <dependency>
      <groupId>org.apache.camel</groupId>
      <artifactId>camel-core</artifactId>
    </dependency>

    <!-- logging -->
    <dependency>
      <groupId>org.apache.logging.log4j</groupId>
      <artifactId>log4j-api</artifactId>
      <scope>runtime</scope>
    </dependency>
    <dependency>
      <groupId>org.apache.logging.log4j</groupId>
      <artifactId>log4j-core</artifactId>
      <scope>runtime</scope>
    </dependency>
    <dependency>
      <groupId>org.apache.logging.log4j</groupId>
      <artifactId>log4j-slf4j-impl</artifactId>
      <scope>runtime</scope>
    </dependency>

    <!-- testing -->
    <dependency>
      <groupId>org.apache.camel</groupId>
      <artifactId>camel-test</artifactId>
      <scope>test</scope>
    </dependency>
  </dependencies>

The example route in camel-archetype-java archetypes Myroutebuilder.java is pretty complex for beginners. Hello world on a timer is generally a more simpler test on to see if things work.

package com.example;

import org.apache.camel.LoggingLevel;
import org.apache.camel.builder.RouteBuilder;

public class MyRouteBuilder extends RouteBuilder {

    public void configure() {

        // from("file:src/data?noop=true")
        //     .choice()
        //         .when(xpath("/person/city = 'London'"))
        //             .log("UK message")
        //             .to("file:target/messages/uk")
        //         .otherwise()
        //             .log("Other message")
        //             .to("file:target/messages/others");

        from("timer:timerName?period=3000")
            .routeId("helloWorldTimer")
            .log(LoggingLevel.INFO, "Hello world");
    }
}

The project generated from archetype comes with exec-maven-plugin which allows you to run the project with mvn exec:java

Java development kit - JDK

If you're using JDK 11 instead of JDK 8 you'll have to modify maven-compiler-plugin configuration a bit.

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-compiler-plugin</artifactId>
    <version>3.8.0</version>
    <configuration>
        <!-- 
        <source>1.8</source>
        <target>1.8</target> 
        -->
        <source>11</source>
        <target>11</target>
    </configuration>
</plugin>

If you've multiple versions of JDK installed you'll have to configure your IDE to use the correct one for the project. With IntelliJ you can configure JDK used by the project from project structure settings.

With VSCode you'll need both JDK 11 and JDK 8 as VSCode Java extensions require JDK 11 to run.

Example settings.json entries for OpenJDK:

"java.configuration.runtimes": [
    {
        "name": "JavaSE-11",
        "path": "C:\\Program Files\\AdoptOpenJDK\\jdk-11.x.x.xxx-hotspot",
        "default": true
    },
    {
        "name": "JavaSE-1.8",
        "path": "C:\\Program Files\\RedHat\\java-1.8.0-openjdk-1.8.0.xxx-x",
    }
],
"java.home": "C:\\Program Files\\AdoptOpenJDK\\jdk-11.x.x.xxx-hotspot"

You might also want to setup your path variable so that java -version returns correct version from the command-line.

Apache Camel: routes inheritance - moving code to the super class

RouteBuilder supports inheritance just like any other class, since onException is RouteBuilder scoped it's fine to define it in a base class. I would however advice against or at least be very careful when defining routes in something like BaseRouteBuilder.

You should also mark BaseRouteBuilder abstract and I would also try to give it more descriptive name like RouteBuilderWithGenericErrorHandler.

RouteBuilders are used to create and add RouteDefinitions to CamelContext and you're not limited to using just one RouteBuilder. Now if you add multiple RouteBuilders to CamelContext that all inherit from BaseRouteBuilder this will cause BaseRouteBuilder.configure to be called multiple times.

This can cause problems like:

RouteId conflicts.
Consumer endpoint URI conflicts.
Same configuration changes getting applied multiple times.
- i.e if you use RouteBuilder.getContext to modify the CamelContext in some way.

In all my route I am calling at the end a http request to create the order and I am using a format to convert it to JSON.

If many of your routes share same routing logic you should split that part to separate route and use it like function. It also makes it easier to add route-scope error handling like what to do when orders api isn't responding or the body provided to direct:createOrder consumer endpoint is invalid.

@Override
public void configure() throws Exception {
    
    from("direct:createOrderA")
        .routeId("createOrderA")
        // .. do stuff for A
        .to("direct:createOrder");
    
    from("direct:createOrderB")
        .routeId("createOrderB")
        // .. do stuff for B
        .to("direct:createOrder");
    
    
    DataFormat format = ...;

    from("direct:createOrder")
        .routeId("createOrder")
        .marshal(format)
        .setHeader(
            Exchange.HTTP_METHOD, 
            constant(org.apache.camel.component.http.HttpMethods.POST)
        )
        .setHeader(Exchange.CONTENT_TYPE, constant("application/json"))
        .to("http://localhost:8080/orders/create-order");
        
}

In Apache Camel what does route() do in a restful declaration?

Using .route allows you to define new route(s) within your rest-definition. It can be handy if your route is short or if you just want to process/transform/validate the message in someway before sending it to your actual consumer endpoint.

For example

rest("/someRoute")
    .id("someRoute")
    .description("Some description")
    .post()
        .consumes("text/plain")
        .produces("text/plain")
        .route()
            .routeId("someRoutePost")
            .process(new SomeMessageProcessor())
            .to("direct:toSomewhere")
        .end()
    .endRest()
    .get()
        .route()
            .routeId("someRouteGet")
            .setHeader(Exchange.HTTP_RESPONSE_CODE, constant(405))
            .setBody(constant("GET not allowed on this route"))
        .end()
    .endRest()

But if you just want to call direct consumer endpoint and do this stuff there instead you can do that.

it is up to ones preference really.

thanks, I see if I wanted to say call .log() I would have to put .route() first

Yes. Camel uses method-chaining with its Java-DSL where something like this is often required. When defining Rest most methods return RestDefinition but if you look closely .route method returns RouteDefition instead.

To get back to RestDefition from route one can use .endRest() as the .end() in the example doesn't really do anything other than make it easier to see where to RouteDefition block ends.

Update: Note that this example is for Camel 3.14.0. In Newer versions of Camel route() and endRest() methods have been Removed from RestDefition class. Example for Camel 3.18.1 can be found here.

Which Enterprise Integration pattern is used in camel

I am facing an issue in finding out which pattern processed the message lastly. Like whether the intercept happened after an aggregate,split or process.After which pattern my intercept processor worked. Is there any way to find out this?

You can get some details from CamelMessageHistory exchange property which is a list of DefaultMessageHistory objects. These contain routeId and some details about the endpoint node like its id. If you're using interceptFrom camel stores the intercepted endpoint uri to CamelInterceptedEndpoint header.

Intercept and print history

intercept()
    .process(new Processor(){

        @Override
        public void process(Exchange exchange) throws Exception {
        
            String output = "History:\n";
            List historyList = (List)exchange.getProperty(Exchange.MESSAGE_HISTORY);
            
            for (int i = 0; i < historyList.size(); i++) {
                
                DefaultMessageHistory messageHistory = (DefaultMessageHistory)historyList.get(i);

                output += "\t[" + (i + 1) + "]" 
                    + messageHistory.getRouteId() +":" 
                    + messageHistory.getNode().getShortName()
                    + "\n";
            }
            System.out.println(output);
        }
    });

This however wont help you much as the information is fairly limited and many patterns like split that create a new exchange with their own new history. However for these you could look if headers like CamelSplitIndex have been set and if it has then determine from that that split has occurred.

if there is any other way to full fill my need to store the details of routing in DB

You could keep things simple and track the state of the exchange using events instead. Just write a processor or component that you can use to write/store what happened and when during the routing. You can use breadcrumbId as unique id for exchange as this persists even after something like split.

The processor or custom component could then simply stream these events to some file or store them to a local database like sqlite for further processing. Avoid sending them straight to external database or service to minimize impact on actual routes.

This kind of diagram is what I need to create from the routes. Thanks in Advance.

Have you looked in to Hawtio? It does a lot of this already. I am running Hawtio with Apache Karaf and it provides me with Route diagrams and fairly detailed profiling data for routes, endpoints and whatnot. Its also open source so you can modify it or use it as reference for your own application.

If you prefer to do something similar yourself you can look in to using JMX to manage and monitor camel application. For my understanding Hawtio uses it under the hood to get more information about applications running inside JVM.

Apache Camel: Aggregator vs Enrich

Thinking Aggregator and Enrich as components can be a bit misleading. They're implementations for enterprise integration patterns in camel and as such very integral to how Apache camel works.

For example pipeline pattern and content enricher are things that you're likely already using all the time with Apache camel without even realizing it. With Apache camel you're almost constantly enriching messages using components, translators, beans and processors.

Aggregation Strategy is used to configure how you want camel to combine one or more exchanges. This means it can be used to combine current exchange to a incoming exchange with enrich or to group multiple exchanges in to one when when using split or aggregate.

How can i know which one to use?

They're completely different patterns, one is for enriching exchanges with new data and one grouping exchanges together.

In terms of defining routes with Camel Java-DSL use of .enrich() is pretty uncommon as generally it's enough to call the endpoint directly with to("uri").

More common is .pollEnrich("uri") as with it one can use consumer endpoints like file, ftp or jms to enrich the exchange. Meaning if you need data from two or more files on a route you can use it to do so.

When it comes to .aggregate() it's mainly used to group exchanges together and handle them as a group after some pre-determined condition has been met like for example every 100 exchanges or after 10 seconds of no new exchanges.

What Exactly Is Apache Camel