Today I was speaking on Coherence SIG event in London.
My topic was "Filtering 100M objects. What can go wrong?". It was a story of solving particular problem and obstacles we have encountered. One noticeable thing about this project - out team was using Performance Test Driven Development approach.
We have started with simplest solution, then were focusing on problem identified by testing.
Slide deck from presentation is below.
Guardian was introduced in Oracle Coherence 3.5 as uniform and reliable mean to detect and report various stalls and hangs on data grid members. In addition to monitoring internal components of Coherence, Guardian has an API accessible for application developer.
While out-of-box Guardian does its job pretty well, there are few aspects you can improve.
There 3 techniques to work with Coherence Guardian. Your can choose to employ all of them or just few.
Guardian is using heartbeat mechanics to detect thread stalls. Internally Coherence code explicitly heartbeat in appropriate points in code. Application code could use similar technique if long execution time is expected. CacheStores are good example of this.
GuardSupport.heartbeat() – sends normal heartbeatGuardSupport.heartbeat(long) – allows you to pass expected time till next heartbeat (e.i. if you expect that SQL query to take several minutes, you could prevent log warning by passing reasonably long timeout before execution SQL statement)Normally the guardian would try to "recover" thread if no heartbeats were received during timeout (eigther specified in configuration or last heartbeat(...) call).
This behavior can be overridden though.
Application can register own Guardable and temporary disable monitoring of current thread.
Below is a code snippet which wraps cache loader operations in Guardable preventing thread interruption
(default way to "recover" worker thread).
public static class GuardianAwareCacheLoader implements CacheLoader {
private CacheLoader loader;
public GuardianAwareCacheLoader(CacheLoader loader) {
this.loader = loader;
}
@Override
public Object load(Object key) {
GuardContext ctx = GuardSupport.getThreadContext();
if (ctx != null) {
KeyLoaderGuard guard = new KeyLoaderGuard(Collections.singleton(key));
GuardContext klg = ctx.getGuardian().guard(guard);
GuardSupport.setThreadContext(klg);
}
try {
return loader.load(key);
}
finally {
if (ctx != null) {
GuardContext klg = GuardSupport.getThreadContext();
GuardSupport.setThreadContext(ctx);
klg.release();
}
}
}
@Override
@SuppressWarnings({ "rawtypes", "unchecked" })
public Map loadAll(Collection keys) {
GuardContext ctx = GuardSupport.getThreadContext();
if (ctx != null) {
KeyLoaderGuard guard = new KeyLoaderGuard(keys);
GuardContext klg = ctx.getGuardian().guard(guard);
GuardSupport.setThreadContext(klg);
// disable current context
ctx.heartbeat(TimeUnit.DAYS.toMillis(365));
}
try {
return loader.loadAll(keys);
}
finally {
if (ctx != null) {
GuardContext klg = GuardSupport.getThreadContext();
GuardSupport.setThreadContext(ctx);
klg.release();
// reenable current context
ctx.heartbeat();
}
}
}
}
public static class KeyLoaderGuard implements Guardable {
Collection<Object> keys;
GuardContext context;
public KeyLoaderGuard(Collection<Object> keys) {
this.keys = keys;
}
@Override
public GuardContext getContext() {
return context;
}
@Override
public void setContext(GuardContext context) {
this.context = context;
}
@Override
public void recover() {
System.out.println("got RECOVER signal");
context.heartbeat();
}
@Override
public void terminate() {
System.out.println("got TERMINATE signal");
}
@Override
public String toString() {
return "KeyLoaderGuard:" + keys;
}
}
Using custom Guardable provides following advantages:
Service failure policy is responsible for reaction on guardian timeouts and critical service failures. Reaction is configurable, but for standalone Coherence processes I prefer to override this policy.
Below is example of service failure policy, which I find more reasonable for dedicated Coherence nodes.
public class ServiceFailureHandler implements ServiceFailurePolicy {
private final static Logger LOGGER = LogManager.getLogger(ServiceFailureHandler.class);
@Override
public void onGuardableRecovery(Guardable guarable, Service service) {
LOGGER.warn("Soft timeout detected. Service: " + service.getInfo().getServiceName() + " Task: " + guarable);
guarable.recover();
}
@Override
public void onGuardableTerminate(Guardable guarable, Service service) {
LOGGER.error("Hard timeout detected. Service: " + service.getInfo().getServiceName()
+ " Task: " + guarable + ". Node will be terminated.");
halt();
}
@Override
public void onServiceFailed(Cluster cluster) {
LOGGER.error("Service failure detected. Node will be terminated.");
halt();
}
private static void halt() {
try {
ThreadUtil.logThreadDump(LOGGER);
LogManager.shutdown();
System.out.flush();
System.err.flush();
} finally {
Runtime.getRuntime().halt(1);
}
}
}
Compared to standard policy it has following advantages:
Integrating you application with Coherence Guardian doesn't require too much code, but could make your logs more clear and troubleshooting less painful. While it will not make your application work faster, it could save hours of digging though logs.
Oracle Coherence data grid has a powerful tool for inplace data manipulation - EntryProcessor. Using entry processor you can get reasonable atomicity guarantees without locks or transactions (and without drastic performance fees associated).
One good example of entry processor would be built-in ConditionalPut processor, which will verify certain condition before overriding value. This, in turn, could be used for implementing optimistic locking and other patterns.
ConditionalPut could accept only one value, but ConditionalPutAll processor is also available. ConditionalPutAll accepts a map of key/values. Using it, we can update multiple cache entries with single call to NamedCache API.
But there is one caveat.
We have placed values for all keys in single map instance inside of entry processor object.
On the other side, in distributed cache keys are distributed across different processes.
How right values would be transferred to right keys?
Answer is simple - every node, owning at least one of keys to be updated, will receive a copy of whole map of values.
In other words, in mid size cluster (i.e. 20 nodes) you may actually transfer 20 times more data over network than really needed.
Modern networks are quite good and you may not notice this traffic amplification effect for some time (as long as you network bandwidth can handle it). But once traffic has reached network limit things are starting to break apart.
Coherence TCMP protocol is very aggressive at grabbing as much of network bandwidth as it can, so other communications protocols will likely perish first.
JDBC connections are likely victim of bandwidth shortage.
Coherence*Extend connection may also suffer (it is using TCP) and proxy nodes may start to fail in unusual ways (e.g. with OutOfMemoryError due transmission backlog overflow).
This problem may be hard to diagnose. TCP is much more vulnerable to bandwidth shortage and you will be kept distracted with TCP communication problems while root cause is excessive TCMP cluster traffic.
Monitoring TCMP statistics (available via MBean) could give you an insight about network bandwidth consumption by TCMP and network health and help to find root cause.
Isolating TCMP in separate switch is also a good practice, BTW
Simple solution is to split keys set by owning nodes, and then invoke entry processor for each subset individually. Coherence API allows you to find node owning particular key.
This approach is far from ideal though:
Another option is relocating your logic from entry processor to trigger and replacing invokeAll() by putAll() (putAll() does not suffer from traffic amplification).
This solution is fairly good and fast, but has certain drawbacks too:
put() is not just put() now),putAll() call),Finally you can use DataSplittingProcessor from CohKit project. This utility class is using virtual cache keys to transfer data associated with keys, then it is using backing map API to access real entries.
This solution has its PROs and CONs too:
In practice I was using all three technique listed above.
Sometimes triggers fit overall cache design quite good.
Sometimes manual data split has its advantages.
And sometimes DataSplittingProcessor is just right remedy for existing entry processors.
Today was speaking at Oracle Coherence SIG at London.
Below you can find slide deck from my presentation.
http://www.slideshare.net/aragozin/coherence-sig-performance-test-driven-development
In this post, I would like to share some knowledge about optimizing indexes in Oracle Coherence.
Normally you should not abuse queering features of your data grid and, hence, you are unlikely to ever need to tune indexing/queering (besides choosing which indexes to create). But sometimes, you really need to squeeze as much performance as you can from your filter based operations. If it is your case, then few tricks described below may be helpful.
You are probably aware of this fact, but it is of critical importance and repeating this one more time will not do any harm. All query planning in Coherence relies on matching (using equals() method) of extractors used in index and filter.
Typical mistakes you could do here:
equals() and hashCode().In all cases above, your code will work, but index will not be used.
Sometimes your query may include criterion for low-cardinality attribute. Not indexing this attribute will cause deserialization of all candidate entries to check attribute value.
Deserialization is something you really want to avoid in Coherence cluster under heavy load. Besides being CPU consuming, deserialization will produce a lot of garbage, risking to bringing you GC out of balance.
Adding index may bring another risk though. If you put your predicates in wrong order, such index may only slow down query.
Below is result of simple benchmark. I was using 2 Coherence storage nodes and 1000000 as data set. Ticker predicate is matching 1000 objects, and side predicate matching 500000. EqualsFilter and AndFilter were used to build query. Execution time of count aggregator was measured.
Tests were run on my laptop, so absolute numbers are not important (and not statistically sound to be honest).
As you can see, if you are unlucky and your query is not in right order, adding index may actually harm query performance.
There is a trick to protect you in this case. NoIndexFilter is a filter wrapper, which disables inverted index lookup for nested index. Forward map of index remains accessible, so testing attribute value will not require desrialization.
As you can see, it takes some toll on "good query", but negates effect of "wrong order of predicates". You can also see that it is still 3 times faster than case where "side" was not indexed.
You can make query above even more faster if you really need to.
Normally, with Coherence, you do not use composite indexes (instead you are indexing attributes individually). Creation of composite index is possible, but you will have to use specially composed queries to exploit composite index.
Code to add composite index will look like
ValueExtractor[] ve = {
new ReflectionExtractor("getTicker"),
new ReflectionExtractor("getSide")
};
MultiExtractor me = new MultiExtractor(ve);
cache.addIndex(me, false, null);
and filter exploiting it will look like
ValueExtractor[] ve = {
new ReflectionExtractor("getTicker"),
new ReflectionExtractor("getSide")
};
MultiExtractor me = new MultiExtractor(ve);
EqualsFilter composite = new EqualsFilter(me, Arrays.asList(ticker, side));
Below are results compared with traditional index/query.
Composite index is awkward to use, but, if it matches your case, you can get significant performance gain.
That is it, for this post. You can also take a look at my slide deck from one of London Coherence SIGs, it explains few more advanced topics about indexes in Oracle Coherence.
Writing automated tests for Coherence based application is quite challenging. Definitely, you can start single node cluster in your test JVM without too much hassle. Unfortunately, single node cluster will not allow you to replicate some important aspects of cluster (a thus, test wont be able to catch a number of nasty bugs).
Running Coherence cluster with several nodes is a bit trickier (Coherence is using singletons). Idea to use classloader to keep multiple Coherence nodes is not new. I’m using this approach for few years already, and …
Today I’m glad to announce availability of my test library for Coherence at Maven central repo.
ChTest is a third generation of my test framework for Oracle Coherence. Besides ability to run multiple Coherence nodes in different classloader, ChTest offers a bunch of extra features.
To name few:
Below is slide deck outlining features of ChTest.
Add following Maven dependency and you would be able to try ChTest yourself.
<dependency>
<groupId>org.gridkit.coherence-tools</groupId>
<artifactId>chtest</artifactId>
<version>0.2.6</version>
<scope>test</scope>
</dependency>
First meet up of Oracle Coherence users in Moscow was held on 14 Mar.
Many thanks to Gene Gleyzer (Oracle), who was key speaker on this event (joining us remotely).
Below are slides from event:
Recently, I was working on renewed version of Coherence data loss listener. New version provides simple utility to attach partition loader to any cache. Such partition loader is guaranteed to be called for every partition of newly created cache or after partition has been lost. Unlike CacheStore, partition loader will be called on the node there it has been added. This way you could have dedicated set loader processes, which are not involved at storing a data. Also it is guaranties that
But let’s get back to a topic of post. I’m actively using JUnit and my test-utils library to automate testing of my Coherence stuff. Test-utils is using class loader isolation to run multiple Coherence nodes in single JVM.
But, unlike many other tests, here I need to test disaster case. Coherence should think that one of its nodes has died. Normally, I’m using CacheFactory.shutdown() to kill virtualized Coherence node, but this way it would be a graceful shutdown.
For data loss listener, I really want to test disaster case.
Naïve approach using timeout is not working well with data grid prioritizing resilience and performance such as Coherence.
What is problem with timeout?
If you let it be too short, there will be too many false positives making grid unstable (JVM may do a GC, OS may start swapping, etc).
If you let it be too long, time of recovery from disaster would be too long.
How this can be improved? Let’s see that kind of disaster could possibly happen with your cluster:
Death of process could be easily tracked if you keep open TCP connection open. OS will close all TCP connections for dead process, so you could make very good assumption that remote process is dead.
Coherence is using so called TCP ring for that purpose. Each cluster node keeps two open TCP connections to other cluster nodes (forming a ring). If cluster detects that both TCP connections have been closed, it has very good reason to disconnect node right now and start recovery procedure.
In case of server/network failure, TCP connection will not be closed immediately. In addition to TCP ring, Coherence is using IP monitor to track reachability of IP addresses. If IP address cannot be reached by rest of nodes, cluster will not hesitate to disconnect all nodes from that IP.
This two tricks allow Coherence to detect real failure very fast, yet to be very tolerant to long GC pauses and other non fatal slowdowns.
In JUnit test all nodes in cluster are sharing same JVM. I cannot really kill a process. To simulate node death, I’m calling Thread.suspend() on all threads related to victim node (a feature of test-utils). This is making node totally unresponsive.
Two mechanisms above should be turned off in Coherence operational configuration. Disconnect timeout also should be set to smaller value (otherwise each test will take too long).
That is it, now I can test disaster cases for Coherence using JUnit.
Below is snippet of actual test:
Here is a link to full java file in SVN.
Client side map listeners are usually added via NamedCache API. They typically receive events from caches hosted on remote JVMs (storage nodes). But regardless of whenever cache event is produced at remote or local JVM, Coherence will deliver it to listeners using dedicated event dispatch thread (or service thread itself for listeners marked as synchronous).
Each cache service has only one event dispatch thread, and it could easily become a bottle neck, limiting speed of cache event processing on client.
Few tips to mitigate this design aspect are below.
Last advice may not be intuitive, but deserialization of map event in Coherence’s event dispatch thread often becomes a bottleneck slowing down event processing rate.
There is a marker interface
SynchronousListener
in com.tangosol.util
package.
You could implement it in your map listener.
But this wouldn’t make map event delivery to your listener synchronous with cache
operation (as you may think), instead it would affect in which thread your
listener is invoked.
Normal listeners are invoked in event dispatch thread.
“Synchronous” listeners would be invoked in service thread
What are the differences?
But let me stress it again, for any type of listener events are delivered asynchronously relative to other cluster nodes.
public static void preloadValuesViaReadThrough(Set<BinaryEntry> entries) {
CacheMap backingMap = null;
Set<Object> keys = new HashSet<Object>();
for (BinaryEntry entry : entries) {
if (backingMap == null) {
backingMap = (CacheMap) entry.getBackingMapContext().getBackingMap();
}
if (!entry.isPresent()) {
keys.add(entry.getBinaryKey());
}
}
backingMap.getAll(keys);
}
public class ExtendConnection {
private static Logger LOGGER = Logger.getLogger(ExtendConnection.class.getName());
private static AtomicInteger CONNECTION_COUNTER = new AtomicInteger();
private int connectionId = CONNECTION_COUNTER.incrementAndGet();
private ConfigurableCacheFactory cacheFactory;
private ConcurrentMap<Service, Service> activeServices = new ConcurrentHashMap<Service, Service>(4, 0.5f, 2);
public ExtendConnection(String configFile) {
cacheFactory = initPrivateCacheFactory(configFile);
}
private DefaultConfigurableCacheFactory initPrivateCacheFactory(String configFile) {
LOGGER.info("New Extend connection #" + connectionId + " is going to be created, config: " + configFile);
XmlElement xml = XmlHelper.loadFileOrResource(configFile, "Coherence cache configuration for Extend connection #" + connectionId);
// transforming configuration
XmlElement schemes = xml.getSafeElement("caching-schemes");
for(Object o: schemes.getElementList()) {
XmlElement scheme = (XmlElement) o;
if (isRemoteScheme(scheme)) {
String name = scheme.getSafeElement("service-name").getString();
if (name != null) {
String nname = name + "-" + connectionId;
scheme.getElement("service-name").setString(nname);
}
}
}
DefaultConfigurableCacheFactory factory = new DefaultConfigurableCacheFactory(xml);
return factory;
}
private boolean isRemoteScheme(XmlElement scheme) {
String name = scheme.getName();
return "remote-cache-scheme".equals(name) || "remote-invocation-scheme".equals(name);
}
public NamedCache getCache(String name) {
NamedCache cache = cacheFactory.ensureCache(name, null);
Service service = cache.getCacheService();
activeServices.putIfAbsent(service, service);
return cache;
}
public InvocationService getInvocationService(String serviceName) {
InvocationService service = (InvocationService) cacheFactory.ensureService(serviceName + "-" + connectionId);
activeServices.putIfAbsent(service, service);
return service;
}
/**
* Warning: this method is not concurrency safe, you may get to trouble if you are accessing caches of services via this connection during shutdown.
*/
public void disconnect() {
for(Service service: new ArrayList<Service>(activeServices.keySet())) {
try {
if (service.isRunning()) {
service.stop();
}
}
catch(Exception e) {
LOGGER.log(Level.WARNING, "Exception during remote service shutdown", e);
}
}
}
}
my Patch mentioned in this post (RFE-7068625) for JVM garbage collector was accepted into HotSpot JDK code base and available starting from 7u40 version of HotSport JVM from Oracle.
JVM
|
Avg. pause
|
Improvement
|
7u3
|
0.0697
|
0
|
7u3, stride=4k
|
0.045
|
35.4%
|
0.0546
|
21.7%
|
|
Patched OpenJDK 7,
stride=4k
|
0.0284
|
59.3%
|
JVM
|
Avg. pause
|
Improvement
|
7u3
|
0.05
|
0
|
7u3, stride=4k
|
0.0322
|
35.6%
|
<pof-config> <user-type-list> <!-- Include definitions required by Coherence --> <include>coherence-pof-config.xml</include> <!-- You should declare type ID for each thrift class you are going to use in Coherence --> <user-type> <type-id>1000</type-id> <class-name>org.gridkit.sample.MyObject</class-name> <serializer> <class-name>org.gridkit.coherence.utils.thift.ThriftPofSerializer</class-name> </serializer> </user-type> ... <!-- Usual POF declarion for application non-thrift classes --> <user-type> <type-id>1100</type-id> <class-name>org.gridkit.coherence.sample.SampleEntryProcessor</class-name> </user-type> </user-type-list> </pof-config>
public class ThriftPofSerializer implements PofSerializer {
private Constructor<?> constructor;
private TSerializer serializer;
private TDeserializer deserializer;
public ThriftPofSerializer(int typeId, Class type) {
try {
this.constructor = type.getConstructor();
this.constructor.setAccessible(true);
this.serializer = new TSerializer();
this.deserializer = new TDeserializer();
} catch (Exception e) {
throw new RuntimeException(e);
}
}
@Override
@SuppressWarnings("rawtypes")
public void serialize(PofWriter out, Object obj) throws IOException {
TBase tobj = (TBase) obj;
byte[] data;
try {
data = serializer.serialize(tobj);
} catch (TException e) {
throw new IOException(e);
}
out.writeBinary(0, new Binary(data));
}
@Override
@SuppressWarnings("rawtypes")
public Object deserialize(PofReader in) throws IOException {
try {
byte[] data = in.readByteArray(0);
TBase stub = (TBase) constructor.newInstance();
deserializer.deserialize(stub, data);
return stub;
} catch (Exception e) {
throw new IOException(e);
}
}
}
Coherence data grid is working with objects (storing, queering, aggregating etc). Java objects are native for Coherence but .NET and C++ are also supported. Usually this is good thing, but sometimes it may cause you problems.
Typically Coherence deployed as dedicated storage cluster (few JVMs over few servers contributing memory resources) with application processes connecting either as storage disabled members or Coherence*Extend clients. It also possible (and fairly often) that storage cluster can be used by multiple applications.
Idea to have separated release/deploy cycle for storage nodes and application looks very attractive. But there is a trick. Classes for objects stored in Coherence distributed cache should be present in classpath of storage nodes. Bummer.
Well, while statement reflects experience of many Coherence
users, it is not technically true. Let me elaborate.
- Coherence storage nodes are storing binary form of keys and values in memory,
- queries (and indexes) may trigger deserialization of object on
server side, but if you stick with POF extractors objects wont’t be deserialized,
- entry processors and aggregator may force objects to be
desirialized on server side, but you can avoid it by using BinaryEntry
API.
So if you are careful, you can get rid of domain classes in classpath on storage nodes. This is huge for complex Coherence based application, you now can just keep grid online while deploying application releases. Of cause, custom entry processor, aggregators, value extractor etc are still have to be available in classpath if you use them, but even if you use them, such kind of code are tending to be much more stable.
Ok, in theory this is achievable, but in practice, it is very hard to achieve. Sticking with binary API to manipulate java object is awkward (and not always efficient due to gaps between object and binary API).
Key idea - use different serializers on client and server side. From same
binary presentation; on client side object is fully deserialized, but on server
side just outer layer and few fields are, most of object data are still binary
blob. This way, on server side we do not need domain classes.
Trick is possible thanks to PofReader.readerRemainder() / PofWriter.writeRemainder(). These methods allow parsing just start of POF stream and keeping its remainder unparsed. At the same time, POF stream are intact, POF extractors can access any attribute of object.
Then is design Coherence based solution, I’m doing my best to keep it modular. Usually there is a layer around Coherence offering application specific, yet generic service. At least one reason to do it in such way - this service can be mocked for testing. Domain object are rarely stored directly in Coherence instead they are wrapped in envelops. Example above illustrates visioned storage, envelop is used to annotate data with timestamp. Envelop is a part of service, while payload of envelop is not.
package example;
import java.io.IOException;
import com.tangosol.io.pof.PofReader;
import com.tangosol.io.pof.PofSerializer;
import com.tangosol.io.pof.PofWriter;
import com.tangosol.util.Binary;
public class Envelop {
public static final int TIMESTAMP_POF = 1;
public static final int DELETED_POF = 2;
public static final int PAYLOAD_POF = 20;
protected long timestamp;
protected boolean deleted;
protected Object payload;
protected Binary binaryPayload;
transient boolean serverMode;
/** TO BE USED WITH SERIALIZER */
protected Envelop(long timestamp, boolean deleted, Object payload, Binary binaryPayload, boolean serverMode) {
this.timestamp = timestamp;
this.deleted = deleted;
this.payload = payload;
this.binaryPayload = binaryPayload;
this.serverMode = serverMode;
}
/** Constructor used on client side */
public Envelop(Object payload, long timestamp, boolean deleted) {
this.payload = payload;
this.timestamp = timestamp;
this.deleted = deleted;
this.serverMode = false;
}
public Object getPayload() {
return payload;
}
public Binary getBinaryPayload() {
return binaryPayload;
}
public long getTimestamp() {
return timestamp;
}
public void setTimestamp(long timestamp) {
this.timestamp = timestamp;
}
public boolean isDeleted() {
return deleted;
}
public void setDeleted(boolean deleted) {
this.deleted = deleted;
}
public static class ServerSerializer implements PofSerializer {
@Override
public Object deserialize(PofReader in) throws IOException {
long timestamp = in.readLong(TIMESTAMP_POF);
boolean deleted = in.readBoolean(DELETED_POF);
Binary data = in.readRemainder();
Envelop dv = new Envelop(timestamp, deleted, null, data, true);
return dv;
}
@Override
public void serialize(PofWriter out, Object o) throws IOException {
Envelop dv = (Envelop) o;
if (!dv.serverMode) {
throw new IllegalArgumentException("Object is in client mode, but server serializer is used. Something wrong with POF config!");
}
out.writeLong(TIMESTAMP_POF, dv.getTimestamp());
out.writeBoolean(DELETED_POF, dv.isDeleted());
out.writeRemainder(dv.getBinaryPayload());
}
}
public static class ClientSerializer implements PofSerializer {
@Override
public Object deserialize(PofReader in) throws IOException {
long timestamp = in.readLong(TIMESTAMP_POF);
boolean deleted = in.readBoolean(DELETED_POF);
Object payload = in.readObject(PAYLOAD_POF);
Binary data = in.readRemainder();
Envelop dv = new Envelop(timestamp, deleted, payload, data, false);
return dv;
}
@Override
public void serialize(PofWriter out, Object o) throws IOException {
Envelop dv = (Envelop) o;
if (dv.serverMode) {
throw new IllegalArgumentException("Object is in server mode, but client serializer is used. Something wrong with POF config!");
}
out.writeLong(TIMESTAMP_POF, dv.getTimestamp());
out.writeBoolean(DELETED_POF, dv.isDeleted());
out.writeObject(PAYLOAD_POF, dv.getPayload());
out.writeRemainder(dv.getBinaryPayload());
}
}
}
Using this technique it is possible to exclude application specific classes from classpath of cluster member JVMs. If you are using .NET or C++ you can even avoid implementing domain objects in Java at all, yet be able to do complex operations using POF extractors.
Posts
All Comments
