Asymmetric read/write path – a trend for scalable architecture

A few years ago I was blogging about architecture using composition of different storage technologies for queering and persistence of data. Rationale behind this was further explained in other my post.

Below is a sketch of such composition:

Unlike classical Von Neumann's memory hierarchy, this composition is asymmetric in terms of read and write paths.

I’m glad to see similar ideas implemented in commercial products marking a trend. In this post I want draw your attention to two very interesting middleware products having principle of composition of specialized storages in their core.

CloudTran

CloudTran is very ambitious product promising performance and scalability for a wide class of application without much extra effort.

CloudTran leverages Oracle Coherence and its integration with EclipseLink to build scalable applications using JPA for persistence. Coherence + EclipseLink (TopLink Grid) is already capable of executing JPQL queries in cache instead of database using rich querying capabilities of Coherence. Missing piece in this tandem was transaction support.

CloudTran is filling this gap adding specialized component for managing transactions. Durability of CloudTran’s transactions is provided by write-ahead disk log (many RDBMSes are using same technique). But unlike RDBMS, CloudTran’s log is not limited to single disk/server, it is distributed (same way as data in Coherence grid) and can benefit from throughput of dozens of disks in cluster.

Full picture of CouldTran based solution is triangle of technologies:

Coherence for fast data retrieval,

CloudTran transaction log for fast and durable transaction persistence,

backend database (relational or NoSQL) is a system of record and long term storage.

Backend database being updated asynchronously is on critical path for neither read nor write, thus database is not limiting application performance. On other side data in Coherence are updated synchronously, so application logic can enjoy strong consistency and ACID transactions (which is huge, eventual constancy is a lot of pain for typical enterprise application with sophisticated data model).

Datomic

Datomic is another young and interesting product promising combination of ACID and scalability. Datomic is also featuring triangle of technologies, but it has own implementation for both in-memory database/cache and transaction persistence (that component is called transactor in Datomic). For system of record you can use either RDBMS or NoSQL storage (Amazon Dynamo). Datomic is offering own API (and own unique approach) for working with data. Datomic is highly influenced by functional paradigm, which would probably make porting existing applications to Datomic non trivial, but for new projects idea of simple but scalable platform featuring ACID data manipulation may be attractive.

Cool toys for enterprise developers

I’m very glad to see such innovative products addressing scalability in not-so-fancy class of enterprise application (of cause both products are not limited to enterprise). IMHO there are enough clones of Google’s BigTable and Amazon’s dynamo in this world already. People working on inventories, reservation systems and other enterprisy stuff also need cool distributed toys.

I’m a little skeptical about future for these products (goals they have set for themselves are just too challenging), but I sincerely which luck to both projects.

Please prove my skepticism wrong ;)

Data Grid Pattern - Proactive caching

Classic and most widely used approach for caching is read through pattern. Look up in cache, then try to load from primary data source if entry is missing in cache - that is how it works. This pattern is easy to implement but it has few unpleasant limitations:

• caching may reduce average response time, but maximum response time is still bound to back end data source response time,

• cache may have stale data, expiry policy may relive this problem to some extent, but aggressive expiry is drastically reducing performance gain from caching.

All in memory pattern

Caching concept is close relative to memory hierarchy principle. Memory hierarchy is one of cornerstones of Von Neumann architecture relies on fact that we have different kinds (in terms of capacity and performance) of memory in system.  With modern hardware dynamic memory capacity is often large enough to keep whole dataset. All-in-memory is term used to describe caching architecture there you have 100% of your data in cache at all times. Having all data in cache allow you to guaranty that no request will have to hit slow backend data source and thus provide more aggressive SLA for max response time. It also may be required for workloads with highly random access to data, there traditional assumptions like 80/20 are not working.

While all-in-memory approach is definitely win in terms of performance, its implementation has few serious challenges:

•  cache should have enough capacity to hold 100% of our data,
• cache should be fault tolerant (losing a portion of data in cache will render it defunct until missing data would be reloaded),
• preloading procedure is often non-trivial due to scale of data set,
• cache should be kept in sync with backend data source.

Capacity and fault tolerance are provided by modern distributed caches out of box, but preloading procedures and cache update strategy are very application specific and fairly challenging to implement.

Below are few practical approaches for keeping cache in synch with primary data source.

Refresh ahead

This is approach similar to expiry policy, but instead of invalidating data, cache proactively refreshing them from master source. Refresh ahead pattern is quite simple, but not very practical though. If cached data set is large (and we are talking about all-in-memory pattern) automatic refreshing is like to overwhelm backend with requests. And even if data set is reasonably small we still have to use fairly long expiry time to make it practical.

So if you looking for all-in-memory cache, refresh ahead are unlikely to help you.

Proactive caching

In contrast with traditional (I would say reactive) caching, with proactive caching pattern you insert/update value in cache as soon as it is updated in backend data source, not at the moment data was requested from cache.

Proactive caching is not necessary should be used with all-in-memory pattern, but combination of these two is very powerful.

Polling updates from DB

An evolutionary step from refresh ahead to proactive caching, would be polling changes for database. Some daemon component should periodically (and frequently) poll database and fetch changes since last cache update. Sounds simple but you have to come out with a way how to "fetch changes since last cache update".  Usually some kind of timestamp is used - each record in table to be cached has a kind of last modified field. Another few challenges:

·         what if cache is representing a query result, not just a single table,

·         'last modified' field should be indexed, otherwise frequent polls will bring database to its knees,

·         polling daemon should be fault tolerant,

·         cache timestamp should be stored somewhere on cache side.

So, implementing this approach will require some amount of work (on both sides, database and cache), but in the end you will have very robust solution.

Only sever limitation of this approach is lag between changes in database and cache, which is no less than poll period.

Long poll

If database has some means for wait/notification in its query language, you can use long poll pattern. 

Using long poll will reduce load on your database server and probably reduce lag between cache and database. Disadvantages of this approach: more code on database side and need to use dedicated thread(s) for polling (because thread will be blocked waiting notification of database side for most time).

In Oracle database long poll could be implemented using DBMS_ALERT package. Though use of DBMS_ALERT may cause serialization of update transaction and harms database performance.

Database notifications

Some databases can push data change notifications directly to clients, without need for polling. E.g. Oracle database has DCN (data change notification) mechanism. Using DCN you can register callbacks which would be notified that certain data have been changed in database. Notification mechanism has few advantages over polling

• less load of database while no data is actually changing,
• smaller lag between changing data in database are reaction in cache.

Notifications approach have disadvantages also

• API usually more complicated, you have to learn more quirks to make it work,
• connection hang problems - application is listening to event on connection which is defunct for some reason,
• notifications may be lost in transition for some reason.

One particular problem with using Oracle DCN in java, was leaking of subscriptions. DCN subscription is remaining active on database side after termination java process (unless it was deregistered explicitly) and eventually you are going to hit limit for active subscriptions. Of cause you should deregister subscription before terminating of client process, but you cannot always guaranty graceful shutdown in practice.

Hooking into database replication

All mature databases have replication feature and thus replication wire protocol. Sometimes, they also provide API to hook into replication channel and programmatically receive all updates (essentially change notifications).

Replication is implemented differently in various databases (or even with different replication solution for same database). But general idea is to make cache act as replication slave for its source database.

Compared to polling or data change notifications, using of replication usually requires more effort from DBA side (they should setup replication slave of master database). It may also cost you some in licensing fees dependent on your replication solution.

MySQL slave protocol does not require any setup on master, so replication links can be created ad hoc. But MySQL has another catch, you should setup row based replication on master database unless you want to parse and execute SQL statements in your cache.

Single cache in front of multiple sources

In large scale system, primary data source may be distrusted itself  (e.g. using sharded database). Having single read through cache may be a problem in this case (doing read through, you have to know which shard to consult about data missing in cache), but with proactive caching such setup it much straightforward. While in read through caching, cache responsibilities of serving requests and acquiring data are coupled. With proactive caching, responsibilities of storing data/serving read requests and feeding cache with data updates may be separated. This way, you can have single cache instance and multiple other components pushing data into it (e.g. each sharing could push its data in single cache).

In this role cache can be though as a kind "materialized view" based up on data in primary (potentially distributed) data source.

Few more links

·         Coherence SIG: Using Oracle Coherence to Enable Database Partitioning and DC Level Fault Tolerance

Using Database Change Notification (DCN) with a Coherence Cache

Indexes: RDBMS vs Coherence vs Lucene

Techincal article comparing popular indexing algorithms between RDBMS, data grid (Oracle Coherence) and full text search engine (Lucene).

Most people are familiar with concept of “index” in SQL database. But indexes are widely used far beyond relational databases. There are even dedicated products, like search engines, which are specializing in indexing of data stored elsewhere. This article will give a high level comparison of capabilities and performance aspects of typical indexes in three different types of technologies: SQL database, in-memory-data-grid (Oracle Coherence) and full text search engine (Apache Lucene).

Full text of article is available at GridDynamics blog - http://blog.griddynamics.com/2011/04/indexes-rdbms-vs-coherence-vs-lucene.html

Data Grid Pattern - Snowflake data schema

Evil of distributed joins

Traditionally data grids based solutions are using denormalized data models. Denormalized model allows reducing number of data lookups in storage and achieve low response time. Traditional normalized data models are no good for distributed key/value storages. Main argument against them is requirement to do multiple joins during typical data access operation. If your data are partitioned, joins are becoming prohibitively expensive. You have to join each partition from left join side with each partition on right side thus cost of join grows in quadratic proportion of your number of partitions. 

Even if you can narrow record set before actual join, your data is still may be located on different servers. Each join between partitioned tables will force your data to be moved across network, adding network latency to response time. More joins – more latency accumulated and you will find you out of your allowed response time very soon.
I other words, generally, in grid, you cannot use neither joins, nor normalized data model (though there may always be an exception).

Denormalization is evil

But data denormalization approach is far from perfect. It has its downsides and serious ones. Data redundancy is a side effect of denormalization. Redundancy increase memory consumption and cost of updates. Loosing consistency due to redundancy is another foe. It is a constant battle.

Data warehousing experience

Data warehousing industry has similar challenges to solve. They have huge amounts of data and also have to deal with distributed data storages. And they have an answer to this challenge for some time!

Analytic databases are usually using "snowflake" data model. Snowflake schema has single fact table (in center of schema) and several dimension tables. Size of fact table is huge; sizes of dimension tables are relatively small.

Below are few examples to illustrate snowflake schema.

From financial industry

and from retail

Snowflake data model allows implementing joins across distributed data storage in smart way. While we still want to have fact data to be partitioned across cluster, we can have a full copy of dimension tables on each node executing queries. In other words we may use replication strategy for small dimension table while keeping large fact data partitioned.
Now we can partially execute query using dimensions which are available in local memory of process and then issue single query to fact table. As long as you do not need to join “facts” table with itself of another facts table query execution requires just one network round trip (and if you really need joins between facts wait for next article “Map/Reduce in data grid”).

How can we you utilize snowflake schema with grid middleware?

First we have to decide how we are going to store dimensions tables in memory. We can use in-memory data grid itself or we can use some in-memory database like Hypersonic or H2 database. Grid will support replication of dimension data out of box and natural support for java object, but in memory relational DB will offer better support for queries (e.g. Oracle Coherence does not support any kind of joins even in replicated storage).
Next question is cluster topology, we may replicate dimension data over all nodes or just few of them (query nodes). These query nodes can be either grid peer or remote grid client.
Functional separation between query nodes and data nodes (one storing partitions of facts table) is probably most practical because they have different memory/CPU usage. This way you can tune JVM memory options and scale each tier independently.

Unfortunately this is just a pattern but not built in feature any data grid product I'm aware of. You still have to complete a lot of non trivial engineering to make it work, but this work worth an effort. This pattern allows to use IMDG in cases they denormalization is unable to solve problem, opening new horizons for using of technology.

Data Grid Pattern - Network shared memory

Using shared memory for inter-process communications is a very popular pattern in UNIX world. Many popular RDBMS (both open source and commercial) are implemented as sets of OS processes communicating via shared memory. Designing a system as a set of functionally specialized processes cooperating with each other helps to keep system more transparent and maintainable; while using shared memory as IPC keeps communication overhead is very low. Shared memory is very efficient compared to other forms of IPC. Several processes can exchange or share information with each other via shared memory without involvement of OS (e.i. no syscalls, no context switching overheads).

Unfortunately shared memory is useful only if all communicating processes are hosted on same server (e.i. connected to same memory circuits). While we cannot use real shared memory when building a distributed system, idea of building system as a set of cooperating specialized processes still remains attractive.

Key point of shared memory is what all process can access and modify same data; all processes have consistent view and can do atomic operations on data (e.g. compare-and-set). Data grid technologies allow us to achieve same features but in network environment. Modern data grid products (e.g. Oracle Coherence, GemStone GemFire, etc) provide us key features for shared-memory-like communication:

• share data with strong consistency guaranties across processes in cluster;
• atomic operations over single key in shared storage.

Still, data grids remain a very different technology. They have with different tradeoffs compared to shared memory. It is impossible to take a PostgresSQL and make it clustered using Oracle Coherence. But we can reuse and extend architectural approach and develop a “network shared memory” pattern as a way to build distributed system.

“Network shared memory” pattern

First you need separate data and processes in your mind. Next you should identify operations with data which are happening in your system. You may have operations such as serving request using data, updating, importing data, exporting data, transforming data, etc. Ideally you should have separate process dedicated for each operation (though it is not always possible due to various reasons, so be holistic). Once you finished with analyzing your data and designing processes, you can start to work on physical model to store your data in grid. Remember, data for grid should be always modeled with access pattern in mind. Just coping data model from RDBMS may produce disastrous results (data modeling for grid is another large topic).

 Why having multiple processes is better?

Why having multiple processes is better compared e.g. with multiple threads within a single process?

Below are few answers:

• You can distributed your processes between servers (e.g. for optimizing resource utilization)
• You can start/stop processes independently
• You can upgrade processes independently
• You have better failure isolation (e.g. memory leak in one process will not bring down whole system)
• You can use less heap per JVM, and have individual memory options for different processes

Sure there are some drawbacks also. A few of them:

• You will have more JVMs running and thus more overhead of JVM itself.
• All processes in the end have to communicate over network and it is not as fast as using memory shared between threads.

 Advanced usages of “Network shared memory”

Data grids have high availability built in. We can leverage this feature to achieve high availability for our processes. First pattern is a “hot standby”. We may have several instances of same process running (e.g. on different boxes) but only one of them active. In case of active process going down, grid will detect process failure and we can promote one of standbys to be new active. Sounds simple, but this simple approach requires “death detection”, “peer discovery” and “distributed consensus”. Believe me, all of these is not an easy task to implement. Fortunately, data grid already have all of this implemented, you just need to use its API. Data grid also can be used to store internal state of process, this way you can implement failover even for stateful processes.

A next evolutionary step of this approach is a load balancing between processes. Data grid can help coordinating processes by sharing routing table for request, state for stateful operations and/or using distributed locks for controlling access to resources.

Data grid is more than just distributed data store

While data grid technology is primarily designed for working with larger data sets, advanced features of modern data grid products may bring benefits to your system even if all your data fit a memory of single server. Their high availability and distributed coordination features may be invaluable for designing modular distributed solutions.

Data Grid Pattern - Data flow mediator

I want to start series of articles about data grid oriented architectural patterns. First pattern I want to present is a “Data flow mediator”. Let me start with example.

Imagine you have a large ecommerce web application and want to do some real time analysis over user actions. You have a stream of simple event, let’s just say clicks. And you need to do real time aggregation by various dimensions like by user, by product, etc. At large scale this task is quite challenging: number of writes is enormous, different dimensions made shading challenging and business want this analytics as close to real time as possible (say few seconds delay). With or without data grid such will remain challenging, but data grid technology have a strong advantages for such task.

Let me now introduce “data flow mediator” pattern.

In this pattern, data grid is used as buffer between systems which produces events (clicks), and systems which consumes information (real time analysis modules).

From producer point of view:

• Grid provides high and scalable throughput,
• Grid provides reasonable balance between durability/performance/cost. In grid we can store data in memory only, protected by multiple redundant copies.

From consumers’ (RT analysis modules) point of view:

• Advanced data grids (e.g. Coherence, GemFire, etc) provide required queering/aggregation tool (implementing efficient queries by multiple dimensions in grid still an art, but it is doable),
• High and scalable read throughput. Different analysis modules may share same “mediator”

From architect point of view:

• Mediator decouples data producer from data consumers, thus localizing impact of changes for each component,
• Data grid is self managing. Imagine managing DB with 50 shards + HA replication + dynamic adding/removing servers to cluster and you will treasure this feature of data grid.

I have demonstrated this pattern with ecommerce examples, but there are similar use cases in finance and telecom. Key prerequisites for this pattern are:

• Large number of small updates,
• Data loss is not fatal (either we can tolerate it or restore data from somewhere else),
• Large number of read queries,
• Queries are reasonable simple but more complicated than just get by primary key,
• Low response time requirements for both read and write,
• Scale is above than single RDMB can handle.

I hope this article was helpful for you to better understand data grid technology and its use cases.

4 levels of replication technologies

Esse about state of art in replication technologies.

Replication is widely used to achieve various goals in information systems, such as better performance, fault tolerance, backup, etc. Four classes of replication technologies are available. Your decision about which technology to use will depend on the logical presentation of the data you are trying to replicate.

Full text of article is available at GridDynamics blog - http://blog.griddynamics.com/2010/02/4-levels-of-replication-technologies.html

Data storages and read vs write controversy

While working with various high-loaded systems recently, I noticed a paradoxical contradiction in the data models. Let me explain through the following comparisons. First let's think about normalized vs. denormalized data models.

	Normalized	Denormalized
Read	Bad - Queries become complex - Joins and nested selects are slow	Good - Fast queries - No joins - Queries are mostly single index lookup (assuming schema is tailored for application need)
Write	Good - Consistency is easier to keep - No self contraction by schema - Less rows to update	Bad - Potential inconsistency in data - More rows to update (data may be duplicated in several places) - Complex update procedures

Let's continue with the next comparison -- single vs. multiple copies of data.

	Single copy	Multiple copies
Read	Bad - Single copy is bottleneck	Good - Operations can be balanced between copies
Write	Good - No need to keep copies in sync - Single place to update	Bad - Update should performed at every copy - Synchronization and consistency issues - Transactional updates become distributed transactions

In general, the patterns are simple. If we think of our data as a set of facts, we anticipate each fact will become several data records (either though database replicas or denormalization of the model). For write access, dealing with each fact as a single record is much more efficient. The impact of this difference between what is and what should be can mitigated by using the optimal storage technology for a given application.

The picture above is very simple (or even simplistic) but it provides some insight for selecting the proper data storage for your applications (or even specific parts of your application). If you have any comments about this, please reply to this post.

Key/value and document-oriented storages

These are data-storage techniques that don't support joins and they usually break the 1^st normal form. Due to limited query support, the schema is often prepared in such a way that application queries become simple index lookups. This can be a very effective approach, but the flip side of this is usually a duplication of the entity's attributes across several tables (or their analogs). This makes updating of data more expensive. These types of storages also tending to use asynchronous disk operations which is further undermine their value as system of record storage. Lack of ACID properties is also of no help.

Search indexes

Search indexes such as Lucene and Sphynx provide excellent query performance. They are using data structures designed for information retrieval at cost of expensive updates. Search engines also require denormalized data model, further complicating writes.

RDBMS

Relational databases have a strong reliance on normalization of the data model. It is extremely difficult for an RDBMS to be effective for both read and write operations. While an RDBMS will never be as fast or as simple as a key/value hash table because the write operations will always be quite expensive (due to indexes and consistency checks), they can be a good middle ground between key/value hash tables and MQ-based storage.

MQ

It may be surprising to some people that I have included message queues in the same discussion with databases. But persistent queues or publish/subscribe systems with quarantined delivery are similar to databases. Submitting a message is like a write operation and receiving a message is like a read. The important thing about MQs is that they can be very efficient for write operations. Read operations in MQ environments are very limited, but experience has shown that "write-only" data storage can fill an important niche.

Convergence of paradigms

In the early days of relational databases, many people were skeptical about their future. The main argument against them was performance. But RDBMS technology has survived. Vendors have made the indexes faster and the query optimizers smarter. Overall, the performance and reliability have improved significantly. While maintaining a strong position in their niche, they have slowly assimilated key features of competing technologies to expand their appeal.

Materialized views are actually a smart way to get the benefits of denormalized data while keeping the schema normalized when it comes to updating. Message queues are also becoming a part of RDBMS offerings (e.g., Oracle RDBMS has queues at its core and PostgreSQL has production-ready built-in queues used by Skype).

We are living in an interesting time. Networking has led to the exponential growth in the amount of data created by, and use for, applications. Data-storage technologies have to adapt and evolve quickly, and it's fascinating to watch this evolution. Good luck with your data storage!

Cache for read and cache for write

Caches are everywhere. We have multiple layers of caches in CPU, we are caching disk blocks in memory and we have a lot of various caches in our applications. Having performance problems you usually have only two viable options (without throwing in more hardware, of cause) – reduce algorithmic complexity (use indexes, hashes, etc) or cache your hot data. All other optimizations in most cases are limited to dozens of percents in performance gain, while caching or indexing, done right, can improve performance ten fold.

Then we are saying caching, we usually have read operation in mind. Such things like in-memory-data-grids (IMDG) extends our vision a little. In-memory-data-grid makes it is possible to perform complex transaction processing in memory without touching disk or DB. But memory is volatile, and in most cases IMDG is just a facade in front of old, slow but reliable RDBMS. And here between lightning fast IMDG and RDBMS there is a place for a class of product not known to the market yet - transactional write cache.

But let me start from the distance, and explain may opinion step by step.

Architecture 1. Just caching

Read, average time – most operations are served by cache.
Read, max time – in case of cache miss, we should access DB (cache does not decrease max operation time).
Write – write through strategy, time is close to DB operation time (cache does not speed up write operations).

We use data grid for caching read access to database. Cache allows us to reduce average time of read operations (but max read time remains limited by DB read time). For some applications only reducing average time is not enough, they impose strict limitations of max operation time also. So we need to evolve architecture further.

Architecture 2. In memory data storage.

Read – all operations are served by in-memory-data-grid.
Write – write through strategy, time is close to DB operation time (cache does not speed up write operations).

Scheme is very similar to previous case, but now we keep all data in cache (no cache misses any more). Implementation of such cache in much more complex, but there are good products (both commercial and open source) on the market, which can do this hard work for you. Problem with read operation performance solved, but fro write operations database is still a bottle. If your business is low latency transactions processing, this architecture is still not good enough. Both read and write operations should be lightning fast, you just can’t afford to touch database at all.
Let’s see what we can do next to improve performance.

Architecture 3. In memory data processing with asynchronous DB writing.

Read – all operations are served by in-memory-data-grid.
Write – all writes also served in in-memory-data-grid, write behind strategy is used to write changes to DB.

Now both read and write operations are served in memory. Changes are written to database asynchronously in the background. There is a possible lag between state of data in grid and in database, but both data in grid and data in database are consistent. Everything looks very good until you start thinking about disaster recovery. And here you have a problem, there are transactions that have been processed in grid but haven’t been written in database. In case of disaster these transaction will be lost.

Yes, IMDG products usually provides some level of fault tolerance, they usually can survive after losing one or more  servers in the grid (while they can rebalance load from fallen servers to surviving ones), but none of them cannot tolerate restarting of all grid. And no, this is just not enough, you cannot afford to loose a single transaction even if whole datacenter is down. In memory data grids can offer us really fast read access to data (including indexing and querying), but storage with fast and reliable persistent writes is out of their scope.

Hey! They are in-memory-data-grids, and they are really good at that they are intended to do. You can’t expect them to do everything. :)

Here I want to conclude - we need another class of middleware products, which will offer us storage with fast and reliable persistent writes. Such persistent storage will be complimentary to IMDG products and will support their “cold start” recovery scenario.

Architecture 4. IMDG + disk based write cache.

Read – all operations are served by in-memory-data-grid.
Write – write through to disk storage; write behind strategy writes from disk storage to DB.

Read operations are served in memory. Write operations are served by disk based persistent storage, and later  being asynchronously written to database. Disk cache guaranties that after restart, all transactions will be recovered and end up in database. Cost of read operations is same as one in architectures 2 and 3, cost of write operations is limited by disk storage.

That benefits do we get introducing another layer? Why is it better than writing directly to DB?

Well, disk storage is a very specialized component (much simple than RDBMS). It should implement only simple set of operations: transactional writes and bulk read (required for recovery only).  Implementation of such storage can be very simple, yet efficient. Latency of write transaction in such storage will be very close to latency of disk transaction itself.

Ok, I think it is clear now that specialized disk storage is more efficient than database, but still there are open questions.

Why should it be implemented as separate component, shouldn’t we integrate it with data grid (usually IMDG products already provide some disk persistent for overflow)?

For me, it has taken to implement such disk storage solution as IMDG extension to understand, that it should be a separate component or may be event separate product.

IMDG and disk storage address different problems, and they use very different approaches. Disks have greater capacity than memory. IMDG often distribute data across network of nodes to increase capacity of grid, but it does not make sense in case of disk storage (even if capacity of one disk is not enough, disk arrays or SAN can address problem of capacity). Number of nodes (processes) also participating in IMDG and disk storage also different.

It is hard to bring strong arguments, but I'm quite confident in this opinion. Devil is in the details.

Retrospective.

This was not first time I was working with architecture of type 3. I start wondering how we were working without dedicated persistent storage layer before. And I was a little surprised when I understood that messaging middleware can, to some extent, fill the gap of write cache.

Read - all operations are served by data grid.
Write - changes are published on message bus, and later written to database and data grid. Changes become visible only after some delay.

In this case, time of write transaction tied to time of sending message (messaging middleware should guarantee durability of message). Somehow comparable to disk storage (messaging middleware should use disk based log to provide durability of messages). But also there is a considerable lag between the end of write transaction and changes being visible to system.

This architecture has somewhat worse performance characteristics than architecture 4 (both messaging middleware and disk based write cache should use the disk, but write cache are potentially more efficient). But it is built up on already available middleware, reliable and outperforms architecture 2.

Only serious drawback of this architecture compared to 4 is a cold start time (it should recover its state either from database or from messaging server, both ways are considerably slower than just reading state from disk).