22 Nov

YPDNGG: You Probably Don’t Need Golden Gate

Before launching into this, I must give due deference to Mogens Nørgaard’s landmark article, You Probably Don’t Need RAC (YPDNR), available here, but originally published Q3 2003 in IOUG Select Journal.  Mogens showed that you can be a friend of Oracle without always agreeing with everything they do.

At Blue Gecko, many of our remote DBA customers have been asking us about Golden Gate.  In July 2009, Oracle bought Golden Gate Software, just one of several companies that have developed log-based replication mechanisms for Oracle and other databases.  This was one of many major acquisitions by Oracle in 2009, including Sun and Relsys. But unlike most of Oracle’s acquisitions, Golden Gate provides very little new functionality not already available in Oracle Streams. Nevertheless, at OpenWorld 2009, Oracle made a shocking announcement.  They declared that Golden Gate would be the primary replication channel for Oracle, and that development would cease on Streams and related components.

Usually, Oracle watches these little third-party products for good ideas, then implements them independently (and better) on their own in the Oracle kernel, then watches as the little third-party companies fizzle out.  A case in point is direct memory access performance sampling.  Precise Software and several other companies in the early 2000s developed  low-impact performance sampling and visualization products for Oracle based on sampling the SGA periodically from an external program.  In version 10g, Oracle answered them with Active Session History (ASH), which did the same thing but better.  Although ASH required customers to purchase the Diagnostic Pack, it still more or less spelled the downfall of competing products.

But in the case of Golden Gate, Oracle already has a log-based replication technology (Streams) built into the kernel, available for a very reasonable price (free with Enterprise Edition).  The only major components that Streams lacks compared to Golden Gate is the ability to replicate across database platforms (Oracle to MSSQL, MySQL, etc. and vice versa).  Even that capability was clearly around the corner: In 11g, Logical Standby (Data Guard), a technology that uses essentially the same stack of components as Streams, gained cross-platform capabilities.

By 11g, Streams has become a mature and stable product, and is far more scalable and configurable than Golden Gate in many ways.  Streams can mine logs on the source or the target, or even a third system.  Depending on the load profile, you can use a wide variety of configuration choices, including parallelism at almost any point.  Streams also allows customers to choose to enforce transaction order or not.

In contrast, Golden Gate’s parallelism is restricted to the apply side, and in parallel mode, does not have the option of guaranteeing transaction order (it is non-ACID). Golden Gate’s parallel apply splits work up by schema, relying on the assumption that interdependent data at the business process level is confined to a single schema at a time.  In other words, if all the tables reside in one schema, then parallel apply doesn’t work, and if they reside in many schemas, the changes in one schema may be applied out of order vis à vis the changes to the other schemas.

Streams is only one of Oracle’s preexisting features that can compete successfully in specific use cases with Golden Gate.  Even more ancient and time-tested solutions such as advanced replication and remote materialized views remain supported and highly effective, depending on the requirement.

If you look at many of the use cases where our customers have deployed Golden Gate, I find that the simplest and most scalable engineering solution would have been remote fast-refresh materialized views.  Our customers often replicate core look-up data, like exchange rates, inventory levels, and other slowly-changing data between Oracle databases within an enterprise.  For this, Golden Gate is completely unjustified, due to cost and complexity compared to remote materialized views.  If it were a question of heterogeneous (inter database product) replication, I completely understand.  But in the majority of situations where we see Golden Gate in use, it is Oracle to Oracle. Given that, I wonder how it could come to pass that responsible people would recommend and implement a solution for such a requirement involving Golden Gate.  Why would Oracle essentially abandon ten years of development and stabilization on a platform like Streams for a less mature, rudimentary product like Golden Gate? Oracle can’t possibly be asking customers to pay additional license fees for a worse version of a product they already own.

So let’s review…

Streams: Mature, complex, requires engineering, highly configurable, scalable, Oracle-only, free.

Golden Gate: Simple, east to deploy, few configuration options, less scalable, expensive, heterogeneous (inter-RDBMS), might break your data.

For me, the corporate direction with regard to Golden Gate is perplexing and smacks of sales-driven (as opposed to requirements and cost-driven) engineering.  I can only imagine what it must be like for the team at Oracle that built Log Miner and AQ into an impressive suite of options including Streams.

UPDATE

Since I posted this, a colleague inside Oracle reassured me that although the product will bear the name ‘Golden Gate,’ it will incorporate features and capabilities of Streams and Golden Gate into a single suite of functions as versions are released over time.  This means that the declarations from inside Oracle do not represent the abandonment of ten years of development.

4 Responses to “YPDNGG: You Probably Don’t Need Golden Gate”

  1. Chris Spargo 03. Dec, 2010 at 7:09 am #

    GoldenGate may be an option however for those that aren’t on an Enterprise license, but still require replication abilities other than materialized views.

    Moving to an Enterprise Edition just for replication is not worth the money.

  2. Jon Mead 02. Jan, 2011 at 11:21 am #

    This is an interesting read. We do a lot of data warehousing work and as such have experienced streams. I have seen streams not be able to deal with the complexity and volume of Oracle to Oracle replication. I have not implemented Golden Gate anywhere yet, but am hoping it provides an answer to some of these problems, the demos I have seen make it very easy to configure, and the marketing suggests it can. The downside being this is yet unproven and carries a potentially high cost I think the key questions are whether the short-comings we have experienced with replication are specific to Oracle Streams and their configuration, and whether Golden Gate can solve them.

  3. JM 10. Feb, 2011 at 7:51 am #

    I have worked with 3 replication products for Oracle: Streams, Shareplex, and GoldenGate. No matter what product you choose, you need to take care of conflicts, data divergence and exception handling.

    Streams is built on top of many other oracle technologies like Logminer and Advanced queueing. The changes are captured in LCRs and wrapped in AnyData data type! And has to be urapped a the target! So it can’t really support large high volume transactions. A large batch update in a single transaction breaks streams(10g; don’t know about 11g). The load and the overhead sits right in the database!

    Shareplex is a great replication tool. Does a good job of what it is designed for. Very robust. Can handle high volumne transactions. Simple to configure. Sits outside the database. Very little overhead. (By the way, Shareplex was around even before streams came out. Streams idea is same, but implemented pretty poorly compared to shareplex.)

    Golden gate is very similar to Shareplex. It is easy to install (the whole s/w packages as a zip and tar). Configuration is easy too. It has more transformation capabilities than shareplex. It can handle heavy (very heavy) transaction volumne. It is not true that Golden gate can parallelize only on the apply side; it can parallelize the extract(capture) and parellelize the pump (propagation) as well. The biggest advantage is that it is the most heterogeneous replication tools out there. It can replicate from Tanden NSK SQL(s), Oracle, Teradata, MySQL, Sybase and many others, and can interface with JMS Queues and flat files as well. It was originally developed for Tandem NSK servers. Great for data warehousing and integration project in addition to being a great table to table replication tool.

  4. Craig Glendenning 02. Oct, 2012 at 3:59 pm #

    Jeremiah,

    I will have to respectfully disagree with your assessment about Streams vs. Golden Gate. Just to give myself some "street ‘cred" on this issue, I have administered a large GG install with over 20 streams in geographically dispersed regions and our original technology assessment was to use streams back in 2009. I chose GG, and have not regretted it. Perhaps I am biased, due to using the technology so much, but here are some things that I want to share:

    - ACID-compliance: If you have foreign keys, you have no problem with cross-schema compliance – in fact, there are many options with how to deal with this
    - API – it is lush.
    - Tokens. There is no comparison between the simplicity in adding payload to your data in GG vs. Streams
    - manageability – GG now is integrated with Grid Control 11g, so it’s very nice.

    If you have the coin for GG, I recommend at least looking at the reference guide to get a feel for the breadth of configurability in GG before deciding on Streams:

    http://www.oracle.com/technetwork/middleware/goldengate/documentation/index.html

    Craig

Leave a Reply