Loading presentation...

Present Remotely

Send the link below via email or IM

Copy

Present to your audience

Start remote presentation

  • Invited audience members will follow you as you navigate and present
  • People invited to a presentation do not need a Prezi account
  • This link expires 10 minutes after you close the presentation
  • A maximum of 30 users can follow your presentation
  • Learn more about this feature in our knowledge base article

Do you really want to delete this prezi?

Neither you, nor the coeditors you shared it with will be able to recover it again.

DeleteCancel

Make your likes visible on Facebook?

Connect your Facebook account to Prezi and let your likes appear on your timeline.
You can change this under Settings & Account at any time.

No, thanks

Apache Camel Business Patterns

No description
by

Ben ODay

on 10 May 2016

Comments (0)

Please log in to add your comment.

Report abuse

Transcript of Apache Camel Business Patterns

focus on primary goals first
don't snowball it
isolate changes to legacy code
don't over think exception handling up front

keep changes/scope to a minimum
bootstrap Camel within existing app (Spring, etc)
introduce a lightweight container (Tomcat, etc)
run standalone during early stages of dev

incrementally expand scope
add timer routes for batch jobs
add routes to expose legacy code
use bean binding to route enable w/o changes
refactor incrementally...
consider functional use cases
need to manage/monitor at runtime?
JMX to stop/start routes, gather stats
need to view/triage exceptions at runtime?
use JMS error queues for visibility/durability
use JMX to copy/move/delete messages

consider non functional use cases
system level monitoring
JVM stats from JMX
3rd party tools (Nagios, AppDynamics)
error monitoring/handling
Camel onException() -> JMS error queues

monitoring tools
VisualVM (JMX)
ActiveMQ web console (browse queues)
Hawt.io web console (AMQ/Camel plugins)

how to monitor

embedded in existing application
bootstrap CamelContext (Spring, Main API)
standalone
main = new Main();
main.addRouteBuilder(new MyRouteBuilder());
maven (dev mode)
maven-camel-plugin
maven-jetty-plugin
in ActiveMQ
ships with Camel
simple deployment
add JAR to /lib
bootstrapp in activemq.xml
add routes to camel.xml
deployment options...
consider the requirements and options…
propagate errors back to callers?
retry automatically? how many times? what frequency?
send to an error queue? send an email?

best practices
keep it simple
don’t over analyze up front -> build/test/refactor
minimize retries
don’t blindly retry everything
retry only if required and errors are temporary
retries can be expensive (tie up threads)
monitor early and often
don’t rely on watching log files
use a log monitoring tool (Splunk, etc)
roll your own email notifications
use an aggregator to reduce noise
how to handle errors

should be an ongoing effort
starts in early design
continues throughout development
ends with performance and acceptance tests

types of tests
POC tests
standlone tests to validate routing logic
functional route tests
standup your real routes to validate flow
use adviceWith to inject mock endpoints/assertions
route performance tests
use camel-dataset to drive/gather stats
business logic tests
prefer Bean Integration to isolate/test w/o Camel
use a mock framework (Mockito, etc) to isolate
how to test Camel apps
create routes
based on your designed pseudo routes/POC

encapsulate core business logic in Beans
use bean binding to minimize Camel APIs
promotes reusability, independent testability

start high level
coarse grained (readable) routing steps
start with the happy path
add route exception handling later

keep routing rules...in routes
don’t reinvent the wheel in POJO code
don't bury complex rules/routing in POJO code
use ProducerTemplate sparingly


start adding routes...

create RouteBuilder classes
for each logical/functional area
this is more art than science
can be as granular as you want
depends on scope/requirements of your app

scoping considerations
single CamelContext per app
multiple Route Builders
onException handling is RB specific
for inter app communication use
JMS, WS, REST, OSGI services, etc
this is all easy to misuse
start simple, refactor, test, keep it simple
divide routes into groups..
use a Camel archetype to create a project skeleton
assumes you know your deployment container
camel-archetype-java (JAR)
camel-archetype-web (WAR)
camel-archetype-blueprint (BUNDLE JAR)

create POC unit tests
validate any complex pattern/technology integration in the design
start by reviewing the Camel source unit tests
create your own tests
make incremental changes to explore technologies
leverage Camel EIPs/Components as needed
explore new patterns/components until you settle on an approach

create a project and POC tests...

refine the design...
across all use cases
identify routes that have common operations
refactor routes to normalize common operations

consider exception handling requirements
affects the structure and granularity of routes

consider the size/scope of the applications
how granular should deployments be

divide routes into independently deployable groups

break down use cases to…

identify processes
data flows, event triggers, periodic events

categorize processes
synchronous (request/reply)
asynchronous (fire/forget, in-only)
high vs. low volume
small vs. large payloads

identify technology interfaces
file, jms, hdfs, solr, jdbc, etc

create pseudo routes
file -> split -> validate -> save
http -> validate -> save
jms -> validate -> save
analyze use cases...
Spring Integration, Mule ESB
similarities
all open source
support EIPs, routing, components
IDE support, relatively easy deployment
differences
Spring Integration
has a narrower component scope
Mule
CPAL license (more restrictive)
has good proprietary components (SAP, Tibco, Seibel, etc)
Camel
more flexible DSL than Mule (XML, Java, Groovy, Scala)
more components than both
strong dev community
integrates with Apache projects

alternatives frameworks
technology challenges...
complex routing
technology integrations
orchestrating services


types of requirements...
highly event based processes
events | messaging
exposing multiple endpoints
http/jms/file => process
complex runtime management
auto/manual flow control
exception handling
what problems are relevant?
Camel Overview - Getting Started

Camel is...

Apache 2.0 Licensed
an open source message routing framework
a large project
100+ components, 50+ EIP implementations
1000s of configurations
backed by a large developer community
designed to integrate with other Apache projects
ActiveMQ, CXF, Karaf, ServiceMix

Camel is not...

an ESB (see Servicemix, Mule)
a message broker (see ActiveMQ, RabbitMQ)
a runtime container (see Tomcat, Karaf)
Executive Summary
my background...

IT Consultant from San Diego, CA
15 years of consulting
5+ years with Apache technologies
Camel (committer), ActiveMQ, CXF, Karaf

why I use Camel...

it simplifies design/dev...
solves common integration issues -> EIPs
wraps common technologies -> components

it promotes...
patterns and best practices
abstracting tech details to focus on logic
visualizing business logic -> routes
presented by Ben O’Day
Business Patterns

first, the motivation
have specific improvements in mind
be prepared to justify the ROI
more so than a new application
have a clear roadmap defined
how is Camel advancing the cause

next, how invasive the refactoring needs to be
should rarely have to scrap and rebuild
consider what the end product requirements are
perhaps just wrap legacy app with Camel
expose new interfaces to old code
JMS, WebServices, REST, etc.
route enable, rewire existing code

refactoring considerations

in a web container (Tomcat, Jetty, etc)
add a /WEB-INF/web.xml
load the Spring ContextLoaderListener
add camel.xml in contextConfigLocation param
build a war, deploy to /tomcat/webapps

in OSGi (Karaf, ServiceMix, etc)
leverage OSGi's lightweight/explicit classloading
build bundle (jar with an OSGi manifest file)
the maven-bundle-plugin will generate the bundle
more complex and powerful than others deployment options
well suited for
deploying multiple applications in the same JVM
complex deployments (clustering, HA, etc) using Fuse Fabric or Zookeeper

deployment options...
onException
can be route scoped or global (all routes in RouteBuilder)
features: configurable redelivery counts/delays, propagation

onException(Exception.class)
.handled(true).maximumRedeliveries(5).redeliveryDelay(10000)
    .to("activemq:generalErrors");

onException(MyBusinessException.class)
.handled(true).maximumRedeliveries(0)
    .to("activemq:businessErrors");

try-catch-finally
modeled after Java's try/catch/finally
defined within a specific route and applies only to that route

from("direct:start")
     .doTry()
         .process(new ProcessorFail())
         .to("mock:result")
     .doCatch(IOException.class, IllegalStateException.class)
         .to("mock:catch")
     .doFinally()
         .to("mock:finally")
     .end();
route exception handling options...
where to start?
create a diagram to visualize the following:
each independent system or component (box)
each interaction (lines with arrows)
consider relevant requirements
SLAs
downtime, latency
performance
expected throughput (min/avg/max/peak)
reliability
guaranteed delivery, strict ordering
error handling
manual triaging vs. automated

requirements/high level design

low level batch processes
despite camel-exec for sys level commands
better suited for cron/shell scripts (sys admins)

full blown web applications
despite jetty, http, servlet components
there are better options

simple, one off use cases...
no wiring/routing needed
JMS producing or consuming
simple pattern implementation
basic component integration


when Camel may not fit

as with any new technology...
choose your battles wisely
don't add unnecessary complexity
don’t use for just one use case

to management...
create a high level design, discuss alternatives
position Camel
as an integration technology
as a lightweight framework
increases productivity -> reuse/abstraction

to technical team...
create a POC and demo it
highlight code reduction, readability, testability
DRY, convention over configuration

be ready to sell it...
Simple Example - Content Based Router

presentation goals...

quick intro to Camel
when to use it
how to navigate options
components, patterns
common use cases
SDLC with Camel
refactoring legacy apps

overall
how to use Camel to address common business problems
Camel Intro
Why Use Camel
Design
Develop
Test
Deploy
Legacy Apps?
Navigating Options
Lifecycle
[my notes on Camel]
RouteBuilder builder = new RouteBuilder() {
public void configure() {
from("direct:a")
.choice()
.when(header("path").isEqualTo("B"))
.to("direct:b")
.when(header("path").isEqualTo("C"))
.to("direct:c")
.otherwise()
.to("direct:d");
}
};
in Java DSL
<camelContext xmlns="http://camel.apache.org/schema/spring">
<route>
<from uri="direct:a"/>
<choice>
<when>
<xpath>$path = 'B'</xpath>
<to uri="direct:b"/>
</when>
<when>
<xpath>$path = 'C'</xpath>
<to uri="direct:c"/>
</when>
<otherwise>
<to uri="direct:d"/>
</otherwise>
</choice>
</route>
</camelContext>
in Spring XML
Simple Example - Recipient List

in Spring XML
RouteBuilder builder = new RouteBuilder() {
public void configure() {

from("direct:a")
.recipientList(header("endpoints"));
}
};
<route>
<from uri="direct:a"/>
<recipientList>
<header>endpoints</header>
</recipientList>
</route>
Components
What is a Component?
How to Use A Component
Component Categories
Custom Components
Patterns
What is a Camel EIP?
Common
Use Cases

Scheduling Jobs
Aggregating Data
Routing Dynamically
Service Level Agreements
general use
direct, seda, file, bean, log, timer
search, big data, caching
lucene, solr, elasticsearch
hdfs, s3, twitter, couchDB, mongoDB
cache (ehcache), hazelcast
management
jmx, ldap, nagios, splunk, zookeeper
testing/debugging
mock, dataset
integration
http, servlet, cxf, cxfrx, jms
protocols
mina, netty, pop3, imap, smpt, snmp, ftp, ssh, stream, xmpp
simply encapsulates an API...to route enable it
for example
camel-file (supports read/write files)
wraps java.io.File API
from("file:inbound").to("file:outbound")

underlying classes/functions
FileComponent
createEndpoint() -> FileEndpoint
FileEndpoint
createProducer() -> FileProducer
createConsumer() -> FileConsumer
FileConsumer
poll() - read files
FileProducer
process() -> create files
you found a component...now what?
review the spec page, samples, unit tests

include camel-[component] jar in your project
add dependency to your pom.xml
w/o maven, just add jars to your classpath

create a basic unit test
copy/paste from component's tests
sanity test your setup
validate desired config options

integrate your component with your app
is your CamelContex setup?
is your RouteBuilder wired in?
first, consider just creating a Bean or Processor
simpler way to encapsulate/reuse custom logic

otherwise, create your own component
use an archetype to stub it out
mvn archetype:generate
-DarchetypeGroupId=org.apache.camel.archetypes
-DarchetypeArtifactId=camel-archetype-component
-DarchetypeVersion=2.13.0
-DgroupId=myGroupId -DartifactId=myArtifactId
manage this project like any other company project
include the artifact in your application as a dependency
add unit tests to make sure its wired in appropriately
optionally publish it for others to use
create a Jira and submit a patch file
host it on github
first steps...
read Camel In Action
review articles and blogs
download the source code
github.com/apache/camel
discussion forums
camel users, stackoverflow

learn related technologies...
Maven, Spring, JMS, CXF, Karaf, EIPs

build a demo app..
in ActiveMQ (ships with Camel)
add routes to camel.xml and start AMQ
create a new project -> maven archetype
build some unit tests
run app
maven camel plugin (jar)
maven jetty plugin (war)
CamelContext
Routes
Endpoints
Processors
CamelContext -> Routes

CamelContext context = new DefaultCamelContext();

context.addRoutes(new RouteBuilder() {
    public void configure() {
        from("file://inbound").to("file://outbound");
    }
});

camelContext.start();
Q/A
Decouple Processes
Runtime Management
problem
need to ensure external resources aren't overwhelmed
need to not exceed agreed upon frequency of use of a resource

solution
throttler EIP
provides configurable control over route throughput

from("seda:GetDataFromClient")
.throttle(100).timePeriodMillis(60000)
.process(GetDataFromClientProcessor());

aggregator EIP
aggregate messages over time period
send fewer/larger messages (List of message during period)
just the latest message (short-lived data...stock quote)

from("seda:GetDataFromClient")
.aggregate(constant(true), MyListAggregator())
.completionSize(100).completionInterval(60000)
.process(GetDataFromClientProcessor());
problem
need to react to events to control route status at runtime

solution
JMX (manually/APIs)
you can use to invoke a route or control the lifecycle of a route

CamelContext APIs
supports route lifecycle APIs
start, stop, suspend, resume

Route Policies
supports route event hooks
onInit(), onRemove(), onStart(), onStop(), onExchangeBegin(), onExchangeDone()


problem
need to combine related data for processing
or sending to systems in batches
solution
aggregator EIP
group by size/timeout/interval
custom aggregator strategy/predicate, leveldb repo
Increasing Throughput
problem
need to process messages in parallel to increase throughput
need to minimize latency for client requests

solution
multi-threading route configurations

threads(poolSize,maxPoolSize)

parallelProcessing()

concurrent consumers (seda, jms)

decoupling processes (async processing)

seda - in memory queuing

jms - (opt) persistent queueing
problem
need to filter data from some processing
need to route based on message content
need to send to multiple dynamic recipients
solutions
filter
problem
how to schedule periodic processes
how to schedule routes to run during specific times

solutions
timer (run job every 60s)

from("timer://MyJob?fixedRate=true&period=60000")
.process(runJob);

quartz (run job every 5 minutes, 8-5pm, weekdays)

from("quartz://MyJob?cron=0+0/5+8-17+?+*+MON-FRI")
.process(runJob);

Scheduled Route Policies

CronScheduledRoutePolicy policy = new CronScheduledRoutePolicy();
policy.setRouteStartTime("0 * * * * ?"); //start every hour
policy.setRouteStopTime("5 * * * * ?"); //stop at 5 after

from("activemq:jobQueue").routePolicy(policy)
.process(runJob);
problem
need to increase system modularity
need to support multi-threaded processing

solution
seda (in memory BlockingQueue)
from("file://orders")
.to("seda:orderQ");
from("seda:orderQ?concurrentConsumers=5")
.process(orderProcessor);

activemq (durable JMS queues)
from("file://orders")
.to("activemq:orderQ?jmsMessageType=Text");
from("activemq:orderQ?concurrentConsumers=5")
.process(orderProcessor);
Converting Data
problem
need to send messages between processes
that expect different formats

solutions
DataFormat (convert between common formats)

from("direct:start").marshal().string().to("jms://stringQ");
from("direct:start").marshal().json().to("jms:jsonQ");

TypeConverter - convertBodyTo(class)

from("direct:start").convertBodyTo(String.class).to("jms://stringQ");

TypeConverter - getBody(class)

from("direct:start")
.process(new Processor() {
public void process(Exchange exchange) throws Exception {
String payload = exchange.getIn().getBody(String.class);
...
Route Enable POJO Code
problem
how to call existing code in a route without having to make any changes or introduce Camel dependencies

solution
Camel Processor (wrap POJO calls)
*side note - a PDF is available if you are not a Prezi fan: http://bit.ly/1lQpMUe
Exchange (ID, Properties, In Msg, Out Msg)




routes -> DSLs
from(...).to(...);
consumers/producers


EIPs
route enabled integration patterns (filter, router, aggregator)

components
route enabled technologies (file, http, jms, ftp, cxf, hdfs)
Camel Building Blocks...
basic example
Evaluating Components
there are hundreds of components
150+ committed to Camel
many more on github, etc.

how to evaluate components...
consider licensing (AL 2.0 vs ?)
version supported (parent/pom.xml)
supported configuration options
producer vs. consumer options
review/run unit tests

Messaging Bus
problem
need to support broadcast events between apps
need to translate messages between apps

solution
establish a centralized JMS broker (ActiveMQ, etc)
JMS enable applications
camel-jms for more complex integrations
Spring JmsTemplate for trivial integrations
Stomp for non Java clients (Ruby, Perl, Python, PHP)
use queues for
messages consumed once
consumers compete for messages
use topics for
broadcast messages
notify all interested applications of an event

an implementation of an integration pattern
based on the EIP book

designed to be...
a common language
a reusable solution
highly configurable
a route enabled interface (multiple DSLs)

from a code perspective
Definition class (route support)
org.apache.camel.model package
Processor class (core logic)
org.apache.camel.processor package

Splitting Data
problem
need to split delimited or grouped data for processing individually

solution
split new line delimited
from("file:input")
.split(body().tokenize("\n"))
.to(itemProcessor);

split new line delimited (large file)
from("file:input")
.split(body().tokenize("\n"))
.streaming()
.to(itemProcessor);

split XML
from("file:input")
.split()
.tokenizeXML("item")
.streaming()
.to(itemProcessor);
ls ./camel/components
Chaining EIPs together...
things to consider
granularity of routes
message pattern
sync vs. async
threading model
error handling
Basic EIP usage...
all EIP code is in camel-core
add the camel-core dependency to your project
create a RouteBuilder class
add a route from() clause and you're off...
use code completion to see options
IntelliJ, Eclipse, etc


Composed Message Processor
start simple and iterate
start with high level flow
stub out beans/processors
add sub routes for modularity
unit test routes end-to-end
makes refactoring easy
Sending Data To A Route
problem
how to send a message to a route from code
how to initiate a route from code

solution
ProducerTemplate
template = camelContext.createProducerTemplate();
POJO Producing - two options...
content based router
recipient list
dynamic router
refactor event based processes into routes
decouple asynchronous operations
increases scalability, throughput, flexibility
add persistent queues (JMS, etc) if necessary

monitoring
add early to add visibility to your application
JMX/VisualVM - raw stats
hawt.io - html5 console for AMQ, Camel, etc
much easier to improve apps that can be visualized

consider deployment options early
restricted to current container?
how will you scale horizontally?

refactor incrementally...

functional testing routes...
Guaranteed Ordering That Scales
problem
need to process a high volume of messages in parallel
must maintain ordering of related messages
can't process related messages concurrently
mitigate resource locking, etc

solution

ActiveMQ Message Groups




from("direct:sendOrder")
.setHeader("JMSXGroupID", xpath("/accountId"))
.to("activemq:OrderQueue");

from("activemq:OrderQueue?maxConcurrentConsumers=10")
.process(orderProcessor);
performance testing routes...
in Java DSL
POC route testing...
[obligatory camel pics]
2. Spring remoting annotation
1. using Camel annotation
Bean Integration/Binding (call POJO directly)
from("jms:inboundOrders")
.aggregate(header("orderId"), new OrderListAggregator())
.completionSize(100).completionInterval(10000)
.process(new BatchOrderProcessor());

public final class OrderListAggregator
extends AbstractListAggregationStrategy<Order> {

public Order getValue(Exchange exchange) {
return exchange.getIn().getBody(Order.class);
}
}
original routes

file -> split ->
validate -> save
http ->
validate -> save
jms ->
validate -> save

normalized

file -> split ->
process order
http ->
process order
jms ->
process order
process order -> validate -> save

`
*for more info, see http://www.slideshare.net/KaiWaehner/spoilt-for-choice-how-to-choose-the
Full transcript