What’s that? You thought Oracle already supported their flagship software product on the current version of their flagship operating system and kernel? Well that is what most of Blue Gecko’s Remote DBA customers thought too. Too often we have had to break the news to a customer who has already dutifully built out their hosts with Oracle’s latest Linux that they will need to reinstall with OEL 5.x in order to be running a supported combination.

Oracle’s certification site adds a few more details. Although the UEK v.2 has been available since March 13, this certification only applies to OEL6 with UEK v.1. So don’t go upgrading to the latest and greatest UEK yet. Also, this certification appears to be valid only for x86-64 architectures, not i386. My Oracle Support’s Certify tool also only shows the certification as valid only for Oracle databases running patchset 11.2.0.3.

Finally, certain things you would want to have when running Oracle, such as the oracle-validated RPM bundle, are still missing from the Unbreakable Linux Network’s OEL6 Yum repositories.

Our advice is to not go rushing to upgrade yet. Let’s give it a few months for the early adopters to flush out the usual SNAFUs, and then take another look.

Related posts:

1. Oracle Certification vs Oracle Support
2. Blue Gecko Announces Free Oracle Database Performance Reviews
3. Oracle Database Hosting on AWS

Did I ever tell you the makers of RAC
had seven features and named each flashback?

Well they did, and it wasn’t a smart thing to do.
You see, when the customers wanted a clue
as to how to keep data from getting deleted
the RAC folks said “flashback” and customers heeded.

They turned on all seven of those flashback features
Each one was a slightly dissimilar creature.
Some used the UNDO, some used flashback files
Some just renamed tables to bin$ styles.

One was a place you keep things for recovering
Another was just for forensic discovering
With so many features called by the same name
when thinks broke no one knew just which one they should blame.

On a Friday three minutes past seventeen hundred,
users ask, “What’s the deal? Our data’s been plundered.”
“It looks like all names in the customer table,
are now ‘John Q Public,’ a certain mislabel.”

It was Nimrod, an intern fresh from his instruction.
What he thought was just test was really production.
“No Problem,” says Morton, the wise DBA,
“Flashback is on. I’ll restore it today.”

First Morton asked Nimrod what time he committed
his update that had any where clause omitted.
“I ran it at just past noon yesterday lunch,
It ran for so long that I went for a munch.”

By now it had been almost thirty one hours
but Morton knew that he could call on his powers,
and on the mere fact that that the undo_retention
was set to two days, a quite lengthy extension.

But woe! When he tried to engage Flashback Query,
he got “snapshot too old” and it ruined his theory
that undo_retention makes Oracle hold
all undo data no matter how old.

It turns out that there was no undo_guarantee
or autoextend which would also be key
to use all Flashback features reliant on Undo
A realization he would slowly come to.

Now Flashback Transaction and Versions Between
were out of the picture, although unforseen.
But he still had four more kinds of Flashback to try
So he thought them each through, to see if they might fly.

Undropping the table would use flashback syntax,
But that wouldn’t help, since the table was intact.
With database flashback all could be reverted
But subsequent changes would then be subverted

Do you know who was governor of California
When Flashback Data Archive came out and I’ll warn ya,
it will not help poor Morton, though it’s called Total Recall.
He’d have had to enable it for tables to see all.

The last of the flashbacks was the old FRA,
the Flashback Recovery Area they say.
And it’s just a directory where things are kept
Like logs and old backups made while Morton slept.

So after all that there was nary a way
to use Flashback of any sort to save the day.
Old-fashioned LogMiner was what Morton used
To restore all the rows that poor Nimrod abused.

With so many flashbacks and so much confusion
I bet that Oracle regrets the profusion
of so many things that they call the same name.
But now its too late and there’s no one to blame.

Related posts:

1. In-Memory Undo
2. Emergency Database Repair
3. Recovering a Schema From InnoDB .frm Files

One day in 2009, a company in Florida called my cell phone at 2AM. They described their problem as follows:

We have a 32-bit Intel server running Red Hat Enterprise Linux 4 and Oracle Database Enterprise Edition 9.2.0.1. There are four databases ranging in size from 20G to 100G. The storage is EXT3 filesystems on partitions of an Apple Xserv RAID5 array.

We had a power outage yesterday, and the database server powered down and booted back up. Prior to yesterday, it has not rebooted for about one year. We have been running trouble-free for the previous year. Upon reboot, Oracle started automatically, but all of the databases appeared as they did about one year ago. It is like the database hasn’t been saving the changes we have been making for the past year. None of the inserts, updates or deletes made in the past year are present in the databases. We are absolutely flummoxed. Please help!

I logged into the server and it was just as they described. Even the alert log and messages files ended suddenly about one year prior, and picked up again on the day of the most recent reboot. There was no trace of the intervening 12 months of work. The customer was ready to resort to their backups, but wanted to understand the problem before they proceeded. In addition, restoring backups would mean losing the last 24 hours of transactions, since archivelogs had not gone to tape for that long, and they were missing just like everything else from before the most recent reboot.

They weren’t the only ones who were flummoxed. I just sat there thinking, “where do I start?” After some poking around, though, I solved the problem. Any guesses what went wrong here? I’ll post the solution in about a week. No fair posting the solution if I’ve told you this story before!

Related posts:

1. Oracle (finally) announces support for Oracle Database 11gR2 on OEL6
2. Oracle Database Hosting on AWS
3. Oracle Backups to S3

In Oracle, this is easy.  The <code>gv$parameter</code> view has a column, <code>ISDEFAULT</code>, that allows a simple SQL query to show which settings are set to non-default values.

It’s not so easy in SQL Server.  There is a view, master.sys.configurations, but it doesn’t have a way to tell if the setting is default or modified or anything.

That’s why I was pleased to find that Michael Campbell came up with a good solution that hard codes known default values into a script that works for SQL Server 2008 and up.

The style of insert used in the above script doesn’t work for SQL Server versions lower than 2008, so I made slight changes to allow it to work on 2005 and lower.  Here it is, with full attribution to Michael Campbell for developing the underlying script and technique. I can’t guarantee all the default values are valid for earlier versions, but the script runs and shows results for SQL Server 2005.

-- Server Configuration (find any non-standard settings)
--        for SQL Server 2008.
DECLARE @config_defaults TABLE (
    name nvarchar(35),
    default_value sql_variant
)

INSERT INTO @config_defaults (name, default_value) VALUES ('access check cache bucket count',0)
INSERT INTO @config_defaults (name, default_value) VALUES ('access check cache quota',0)
INSERT INTO @config_defaults (name, default_value) VALUES ('Ad Hoc Distributed Queries',0)
INSERT INTO @config_defaults (name, default_value) VALUES ('affinity I/O mask',0)
INSERT INTO @config_defaults (name, default_value) VALUES ('affinity mask',0)
INSERT INTO @config_defaults (name, default_value) VALUES ('affinity64 I/O mask',0)
INSERT INTO @config_defaults (name, default_value) VALUES ('affinity64 mask',0)
INSERT INTO @config_defaults (name, default_value) VALUES ('Agent XPs',1)
INSERT INTO @config_defaults (name, default_value) VALUES ('allow updates',0)
INSERT INTO @config_defaults (name, default_value) VALUES ('awe enabled',0)
INSERT INTO @config_defaults (name, default_value) VALUES ('backup compression default',0)
INSERT INTO @config_defaults (name, default_value) VALUES ('blocked process threshold (s)',0)
INSERT INTO @config_defaults (name, default_value) VALUES ('c2 audit mode',0)
INSERT INTO @config_defaults (name, default_value) VALUES ('clr enabled',0)
INSERT INTO @config_defaults (name, default_value) VALUES ('common criteria compliance enabled',0)
INSERT INTO @config_defaults (name, default_value) VALUES ('cost threshold for parallelism',5)
INSERT INTO @config_defaults (name, default_value) VALUES ('cross db ownership chaining',0)
INSERT INTO @config_defaults (name, default_value) VALUES ('cursor threshold',-1)
INSERT INTO @config_defaults (name, default_value) VALUES ('Database Mail XPs',0)
INSERT INTO @config_defaults (name, default_value) VALUES ('default full-text language',1033)
INSERT INTO @config_defaults (name, default_value) VALUES ('default language',0)
INSERT INTO @config_defaults (name, default_value) VALUES ('default trace enabled',1)
INSERT INTO @config_defaults (name, default_value) VALUES ('disallow results from triggers',0)
INSERT INTO @config_defaults (name, default_value) VALUES ('EKM provider enabled',0)
INSERT INTO @config_defaults (name, default_value) VALUES ('filestream access level',0)
INSERT INTO @config_defaults (name, default_value) VALUES ('fill factor (%)',0)
INSERT INTO @config_defaults (name, default_value) VALUES ('ft crawl bandwidth (max)',100)
INSERT INTO @config_defaults (name, default_value) VALUES ('ft crawl bandwidth (min)',0)
INSERT INTO @config_defaults (name, default_value) VALUES ('ft notify bandwidth (max)',100)
INSERT INTO @config_defaults (name, default_value) VALUES ('ft notify bandwidth (min)',0)
INSERT INTO @config_defaults (name, default_value) VALUES ('index create memory (KB)',0)
INSERT INTO @config_defaults (name, default_value) VALUES ('in-doubt xact resolution',0)
INSERT INTO @config_defaults (name, default_value) VALUES ('lightweight pooling',0)
INSERT INTO @config_defaults (name, default_value) VALUES ('locks',0)
INSERT INTO @config_defaults (name, default_value) VALUES ('max degree of parallelism',0)
INSERT INTO @config_defaults (name, default_value) VALUES ('max full-text crawl range',4)
INSERT INTO @config_defaults (name, default_value) VALUES ('max server memory (MB)',2147483647)
INSERT INTO @config_defaults (name, default_value) VALUES ('max text repl size (B)',65536)
INSERT INTO @config_defaults (name, default_value) VALUES ('max worker threads',0)
INSERT INTO @config_defaults (name, default_value) VALUES ('media retention',0)
INSERT INTO @config_defaults (name, default_value) VALUES ('min memory per query (KB)',1024)
-- NOTE: SQL Server may change the min server
--   memory value 'in flight' in some environments
--    so it may commonly show up as being 'non default'
INSERT INTO @config_defaults (name, default_value) VALUES ('min server memory (MB)',0)
INSERT INTO @config_defaults (name, default_value) VALUES ('nested triggers',1)
INSERT INTO @config_defaults (name, default_value) VALUES ('network packet size (B)',4096)
INSERT INTO @config_defaults (name, default_value) VALUES ('Ole Automation Procedures',0)
INSERT INTO @config_defaults (name, default_value) VALUES ('open objects',0)
INSERT INTO @config_defaults (name, default_value) VALUES ('optimize for ad hoc workloads',0)
INSERT INTO @config_defaults (name, default_value) VALUES ('PH timeout (s)',60)
INSERT INTO @config_defaults (name, default_value) VALUES ('precompute rank',0)
INSERT INTO @config_defaults (name, default_value) VALUES ('priority boost',0)
INSERT INTO @config_defaults (name, default_value) VALUES ('query governor cost limit',0)
INSERT INTO @config_defaults (name, default_value) VALUES ('query wait (s)',-1)
INSERT INTO @config_defaults (name, default_value) VALUES ('recovery interval (min)',0)
INSERT INTO @config_defaults (name, default_value) VALUES ('remote access',1)
INSERT INTO @config_defaults (name, default_value) VALUES ('remote admin connections',0)
INSERT INTO @config_defaults (name, default_value) VALUES ('remote login timeout (s)',20)
INSERT INTO @config_defaults (name, default_value) VALUES ('remote proc trans',0)
INSERT INTO @config_defaults (name, default_value) VALUES ('remote query timeout (s)',600)
INSERT INTO @config_defaults (name, default_value) VALUES ('Replication XPs',0)
INSERT INTO @config_defaults (name, default_value) VALUES ('scan for startup procs',0)
INSERT INTO @config_defaults (name, default_value) VALUES ('server trigger recursion',1)
INSERT INTO @config_defaults (name, default_value) VALUES ('set working set size',0)
INSERT INTO @config_defaults (name, default_value) VALUES ('show advanced options',0)
INSERT INTO @config_defaults (name, default_value) VALUES ('SMO and DMO XPs',1)
INSERT INTO @config_defaults (name, default_value) VALUES ('SQL Mail XPs',0)
INSERT INTO @config_defaults (name, default_value) VALUES ('transform noise words',0)
INSERT INTO @config_defaults (name, default_value) VALUES ('two digit year cutoff',2049)
INSERT INTO @config_defaults (name, default_value) VALUES ('user connections',0)
INSERT INTO @config_defaults (name, default_value) VALUES ('user options',0)
INSERT INTO @config_defaults (name, default_value) VALUES ('xp_cmdshell',0)

SELECT c.name, value, value_in_use, d.default_value
from sys.configurations c
INNER JOIN @config_defaults d ON c.name = d.name
where
    c.value != c.value_in_use
    OR c.value_in_use != d.default_value
go

Related posts:

1. Finding thread IDs and names of SQL Server background threads
2. SQL Server
3. Deriving Optimal Configuration Values Using 11g Database Replay

As with Amazon’s other cloud services, you control and manage RDS services using a web application API.  You can either write your own software to do this, or use Amazon’s command line API tools or Amazon’s web-based console.

RDS Oracle instances are available in a variety of Oracle editions, hardware and storage sizes. For the first time, you can license Oracle by the hour through RDS. Unfortunately, this utility licensing model is limited to RDS instances running Standard Edition One. For the other editions, customers must “bring their own” Oracle licenses.

When you start an RDS Oracle instance you provide the web API with a variety of specifications, including:

• The Oracle edition (Standard Edition One, Standard Edition, Enterprise Edition)
• The license model (Bring-your-own or License-included for SE1)
• The instance class (Small, Large, Extra large, 2x extra large or 4x extra large)
• Automatic minor-version upgrades (yes or no)
• Storage allocation (10G – 1024G)
• Instance ID, database name and listener port
• A username and password for logging in
• A parameter group (a set of Oracle initialization parameters defined via the web API)
• Security groups (what machines can connect)
• Backup retention
• backup and maintenance windows

I fired up an instance of Enterprise Edition with the name bg01 and master user bg just to test drive the product. Once the instance is up and running, Amazon gives you an endpoint address, like bg01.csmmbl5fszl6.us-west-1.rds.amazonaws.com.  If you have added your own IP address to the security group under which you started the RDS instance, the you will be able to connect to your database using SQL*Plus:

sqlplus bg/foobar123@bg01.csmmbl5fszl6.us-west-1.rds.amazonaws.com:3306/bg01

The username you provided on startup has DBA privileges, but they are limited. It can:

• Create users
• Create tablespaces
• Create tables, indexes, and all other usual objects

A full listing of the privileges available to the master user can be obtained using Pete Finnigan’s find_all_privs.sql script.

Notably, DDL triggers are in place to prevent this master user, and other users it might create, from doing certain things that might make it possible to do things outside the scope of the database. The users that RDS customers can access cannot:

• Grant alter system or database
• Grant restricted session
• Grant create or drop any directory
• Grant any privilege
• DATAPUMP_EXP_FULL_DATABASE, DATAPUMP_IMP_FULL_DATABASE, IMP_FULL_DATABASE, EXP_FULL_DATABASE
• DDL on any object owned by SYS
• Alter user, revoke or any DDL on SYS, SYSTEM or RDSADMIN (a special user installed by Amazon)
• Drop tablespace RDSADMIN
• Create public synonyms for objects belonging to SYS, SYSTEM or RDSADMIN

There is currently no support for any implementation of Enterprise Manager (Grid Control or Database Control). Amazon instead brings essential monitoring metrics for the instance into the CloudWatch API, which you can monitor using your own software via the API, or via the RDS web console.

I’ll post more soon on the new Oracle RDS service, but I will leave you with one piece of advice:  If you don’t want to permanently break your Oracle RDS instance, don’t try to use your master user to create a public synonym named v$database.

Related posts:

1. Oracle Processor Licensing on Amazon EC2
2. Oracle E-Business Suite in the Amazon Cloud
3. High Performance Oracle 11g in the Amazon Cloud

Compared to conventional off site tape vaulting services, the advantages of vaulting database and other backups to S3 are many.  S3 backups are always on line, so you never have to wait for a  truck to arrive with your tapes. S3 backups are replicated, so if one of Amazon’s availability zones experiences a failure, your data is still intact and available at one of their other zones. Best of all, Amazon also offers the Elastic Compute Cloud (AKA EC2, virtual server hosts by the hour), so your S3 backups double as a super-low-cost disaster recovery strategy. S3 is low-cost, starting at just 3.7¢ / GB / month for storage, and 10¢ / GB for uploads.

I back up all my home computers to S3 using third-party software called Jungle Disk.  Jungle Disk runs in the background on a Mac or PC and backs up new data to the cloud ever five minutes (or whatever frequency you specify). Many times these backups have come in handy for me, as I am able to browse and retrieve files from my home computers (such as photos and documents) from the office, without my home computers even being on.

Sounds like an ideal backup solution for small to medium-sized business, right?  So what’s the catch?

The catch is your Internet connection.  Consider the following:

A typical cable Internet connection with an actual upload rate of 2.8 Mb/s could transfer a 100G file to Amazon S3 in just over ten hours.  For many businesses, that’s the whole backup window. If there is more than 100G to transfer, you’re out of luck.  If your servers are co-located somewhere with a fast connection to the Internet, you might get better transfer rates, but there are other limiting factors, like number of hops to Amazon S3 and overall latency.

Compress before uploading

If the backup files are not already compressed, you can almost always dramatically improve upload times by compressing them before uploading them.

Parallel uploads save the day

Upload performance to Amazon S3 can almost always be improved by running uploads in parallel.  Choosing a degree of parallelism depends on the connection between your site and Amazon, the host from which you are uploading, and several other factors.  The best way to determine your optimal degree of parallelism is to test it!

Blue Gecko happens to have a customer who wants to vault their Microsoft SQL Server backups to Amazon S3.  Unlike Oracle, SQL Server has no native facility that allows it to stream backups directly to S3 like tape. Instead, with this customer, we will compress and upload their database backups after they complete each night. To make it easy to find the optimal degree of parallelism, I delved into the murky world of Windows command shell programming. Against all instincts, I wrote this tool in Batch so that it would work easily on any of this customer SQL Server hosts.

This tool is designed to allow you to effectively determine the optimal parallel degree for backing up data from a particular server over the Internet to Amazon S3. It generates its own large files to upload.  All you need is an Amazon S3 account.  The tool comes as a pair of scripts that call a Ruby tool called s3cmd. To use it, follow these steps:

1. Download and install Ruby 1.8.7-p334 for Windows.
2. Download S3Sync into a convenient directory.
3. Download and install gnuwin gzip and gnuwin tar for Windows
4. Open a command prompt window, and change to the directory where you downloaded S3Sync

   c:> cd \my_directory

5. Unzip and untar the S3Sync package, then change to the S3Sync directory:

   c:\my_directory> "\Program Files\GnuWin32\bin\gzip.exe" -d s3sync.tar.gz
   c:\my_directory> "\Program Files\GnuWin32\bin\tar.exe" xvf s3sync.tar
   c:\my_directory> cd s3sync
   

6. Edit a file called test_parallel.bat, and paste the following contents into the file:

   echo off
   set /a filesize = %1 / %2
   for /l %%v in (1,1,%2) do (
    fsutil file createnew dummy.%%v %filesize%
    )
   for /l %%v in (1,1,%2) do (
    start /b upload %3 %%v
    )
   echo on
   

7. Create a bucket for testing. Make sure to substitute your AWS security credentials in the appropriate places:

   c:\my_directory> set AWS_ACCESS_KEY=<em>your AWS access key ID</em>
   c:\my_directory> set AWS_SECRET_ACCESS_KEY=<em>your AWS secret access key</em>
   c:\my_directory> set AWS_CALLING_FORMAT=SUBDOMAIN
   c:\my_directory> s3cmd.rb createbucket my_test_bucket_1234
   

8. Edit a file called upload.bat, and paste the following contents into the file. Make sure to substitute your AWS security credentials in the appropriate places:

   echo off
   set AWS_ACCESS_KEY_ID=<em>your AWS access key id</em>
   set AWS_SECRET_ACCESS_KEY=<em>your AWS secret access key</em>
   set AWS_CALLING_FORMAT=SUBDOMAIN
   echo %time%
   s3cmd.rb put %1:dummy.%2 dummy.%2
   echo %time%
   del dummy.%2
   

9. Now you can start testing parallel uploads to AWS. The syntax to call test_parallel.bat is:

   c:\my_directory> test_parallel file_size degree bucket
   

   Here is an upload example for 128M at parallel 10 to a bucket called my_test_bucket_1234:

   c:\my_directory> test_parallel 125829120 10 my_test_bucket_1234
   

   The elapsed time can be determined as follows:

• Look for the first timestamp displayed in the output and note it.
• Waitfor the last timestamp to display. Note it.
• The intervening time is the elapsed time for the upload. You can use Ex el or any other tool you like to calculate time deltas.

I typically run the upload without parallelism (degree = 1), then increase it in increments of five.  If there is any doubt as to which S3 region will provide the best performance, I create a bucket in each region (US-West and US-East), then perform identical tests against each.

Related posts:

1. Amazon Web Services FAQ
2. s3fox does not create valid export manifest files
3. Amazon Web Services Packages

Only those companies that failed to perform rudimentary availability design on their EC2-based systems have experienced prolonged outages as a result of this week’s event.  This is only as would be expected – Amazon.com does not promise to make your application highly available.  What Amazon EC2 provides is a rich set of tools that allows anyone that is serious about building a highly available application to do so on EC2.

This week’s EC2 failures have provided plenty of fodder for the cloud skeptics, as well it should. Cloud skeptics hold EC2 and other cloud services’ feet to the fire, forcing them to address real concerns with the paradigm. What is far more alarming than the told-you-sos from the cloud skeptics is the torrent of media ignorance regarding what cloud computing and EC2 fundamentally provide.

Take this article in the Wall Street Journal for example.  Quoting the authors:

A main issue at the center of this controversy is why Amazon hasn’t been able to re-route capacity between data centers that would have avoided this problem and ensured the websites of its users would still operate properly.

Here the authors seem to be referring to EC2 availability zones.  As most who have worked even a little with EC2 know, when you run an instance or store volumes in one availability zone, there is no automatic mechanism available to “re route capacity” between availability zones. If you want your application to survive the failure of an availability zone, you must implement a high availability contingency. For instance, you can frequently back up (using the EBS snapshot feature) your storage volumes so that you can re-instantiate the system in a surviving availability zone. In this week’s outage, all but one of Amazon’s US East availability zones were functioning normally within about four hours. Only those customers with systems in one of Amazon’s four US-East zones could not reliably access their data in their Elastic Block Store (EBS) volumes. If those customers had simply performed regular backups (snapshots) of their volumes, the outage would have been confined to a few hours, not 40+ hours.

It is hard to blame the media though, when even the customers of EC2 showcase a complete misunderstanding of what they should expect from the infrastructure on which they have built the systems that support their very businesses.  In the same article cited above, Simon Buckingham, CEO of Appitalism is extensively quoted misunderstanding the fundamentals of EC2:

We’re past the point of this being a routine outage… Customers like myself have assumed that if part of Amazon’s data center goes down, then traffic will get transferred in an alternative capacity… The cloud is marketed as being limitless, but what this outage tells us is it’s not.

That is an interesting assumption indeed, Mr. Buckingham. I would assume that the CEO of a web-based company would spend at least enough time understanding his company’s own infrastructure to realize that he should be talking to his own engineers about why the failed to design a robust multi-zone backup solution on EC2, rather than imagining capabilities for EC2 that do not and have never been asserted to have existed.

I hope the upshot of this event will be more comprehensive and careful engineering of solutions deployed to the cloud.  I fear however, that given the tenor of the media coverage and customer reactions, the onus will not be placed where it belongs, on the customers’ own engineers, and instead will only result in undeserved bad press for Amazon.

For what its worth, we at Blue Gecko frequently help our customers deploy robust highly-available solutions on EC2, that would have easily recovered in the four-to-five hour time frame, or even experienced no outage at all, rather than the 40+ hour nightmare affecting some customers of EC2.

Related posts:

1. High Performance Oracle 11g in the Amazon Cloud
2. May 5th, Seattle, Oracle in the Cloud Seminar
3. A Cloud over San Francisco for OpenWorld 2010

Oracle’s pre-RMAN hot backup mode is the subject of one of the most pervasive and persistent misconceptions about Oracle.

During an Oracle tablespace hot backup, you (or your script) puts a tablespace into backup mode, then copies the datafiles to disk or tape, then takes the tablespace out of backup mode. These steps are widely understood by most DBAs.

However, there is a popular misconception that datafiles are “quiesced,” “frozen,” “offlined” or “locked” during backup mode. So many people think it is true, that it appears in some books on Oracle and on numerous websites. Some have even reported that they learned this from DBA class instructors.

The myth has a couple permutations. One is that while the datafiles are allegedly not writable, changes are stored somewhere in the SGA, the redologs, the rollback segments or some combination thereof, then written back into the datafile when the tablespace is taken out of backup mode. There is a passage in the SAMS title Oracle Unleashed describing this supposed mechanism.

When you place a tablespace in backup mode, the Oracle instance notes that a backup is being performed and internally compensates for it. As you know, it is impossible to make an authentic copy of a database file that is being written to. On receipt of the command to begin the backup, however, Oracle ceases to make direct changes to the database file. It uses a complex combination of rollback segments, buffers, redo logs, and archive logs to store the data until the end backup command is received and the database files are brought back in sync. Simplifying a hot backup in this way is tantamount to classifying the USS Nimitz as a boat. The complexity of the actions taken by the Oracle RDBMS under a hot backup could consume an entire chapter and is beyond the scope of this book. What you should understand is the trade-off for taking a hot backup is increased use of rollback segments, redo logs, archive logs, and internal buffer areas within the SGA.

– Oracle Unleashed, Copyright © SAMS/Macmillan, Inc. 1997, chapter 14)

No No No!  Stop making stuff up! Oracle’s tablespace hot backup does not work this way at all. It is actually a simple, elegant and failure-resistant mechanism. It absolutely does not stop writing to the datafiles. It actually allows continued operation of the database almost exactly as during normal operation. Contrary to the characterization as “complex” in SAMS Oracle Unleashed, it can be  summarized in a few steps:

• DBWn checkpoints the tablespace (writes out all dirty blocks as of a given SCN)
• CKPT stops updating the Checkpoint SCN field in the datafile headers and begins updating the Hot Backup Checkpoint SCN field instead
• LGWR begins logging full images of changed blocks the first time a block is changed after being written by DBWn

Those three actions are all that is required to guarantee consistency once the file is restored and recovery is applied. By freezing the Checkpoint SCN, any subsequent recovery on that backup copy of the file will know that it must commence at that SCN. Having an old SCN in the file header tells recovery that the file is an old one, and that it should look for the archivelog containing that SCN, and apply recovery starting there.

Note that during hot backup mode, checkpoints to datafiles are not suppressed. Only the main Checkpoint SCN flag is frozen, but CKPT continues to update a Hot Backup Checkpoint SCN in the file header.

There is a confusing side effect of having the Checkpoint SCN frozen at an SCN earlier than the true checkpointed SCN of the database. In the event of a system crash or a shutdown abort during hot backup of a tablespace, the automatic crash recovery routine during startup will look at the file headers, think that the files for that tablespace are out of date, and will suggest that you need to apply old archived redologs in order to bring them back into sync with the rest of the database. Fortunately, no media recovery is necessary. With the database started up in mount mode:

SQL> alter database end backup;

This action will bring the Checkpoint SCN in the file headers in sync with the Hot Backup Checkpoint SCN (which is a true representation of the last SCN to which the datafile is checkpointed). Once you do this, normal crash recovery can proceed during ‘alter database open;’.

By initially checkpointing the datafiles that comprise the tablespace and logging full block images to redo, Oracle guarantees that any blocks changed in the datafile while in hot backup mode will also be present in the archivelogs in case they are ever used for a recovery. Most of the Oracle user community knows that  Oracle generates a greater volume of redo during hot backup mode. This is the result of Oracle logging of full images of changed blocks in these tablespaces. Normally, Oracle writes a change vector to the redologs for every change, but it does not write the whole image of the database block. Full block image logging during backup eliminates the possibility that the backup will contain unresolvable split blocks. To understand this reasoning, you must first understand what a split block is.

Typically, Oracle database blocks are a multiple of O/S blocks. For example, most Unix filesystems have a default block size of 512 bytes, while Oracle’s default block size is 8k. This means that the filesystem stores data in 512 byte chunks, while Oracle performs reads and writes in 8k chunks or multiples thereof. While backing up a datafile, your backup script makes a copy of the datafile from the filesystem, using O/S utilities such as copy, dd, cpio, or OCOPY. As it is making this copy, your process is reading in O/S-block-sized increments. If DBWn happens to be writing a DB block into the datafile at the same moment that your script is reading that block’s constituent O/S blocks, your copy of the DB block could contain some O/S blocks from before the database performed the write, and some from after. This would be a split block. By logging the full block image of the changed block to the redologs, Oracle guarantees that in the event of a recovery, any split blocks that might be in the backup copy of the datafile will be resolved by overlaying them with the full legitimate image of the block from the archivelogs. Upon completion of a recovery, any blocks that got copied in a split state into the backup will have been resolved by overlaying them with the block images from the archivelogs.

All of these mechanisms exist for the benefit of the backup copy of the files and any future recovery. They have very little effect on the current datafiles and the database being backed up. Throughout the backup, server processes read datafiles DBWn writes them, just as when a backup is not taking place. The only difference in the open database files is the frozen Checkpoint SCN, and the active Hot Backup Checkopint SCN. To demonstrate the principle, we can formulate a simple proof:

Create a table and insert a row:

SQL> create table fruit (kind varchar2(32)) tablespace users;
Table created.

SQL> insert into fruit values ('orange');
1 row created.

SQL> commit;
Commit complete.

Force a checkpoint, to flush dirty blocks to the datafiles.

SQL> alter system checkpoint;
System altered.

Get  the file name and block number where the row resides:

SQL> select dbms_rowid.rowid_relative_fno(rowid) file_num,
            dbms_rowid.rowid_block_number(rowid) block_num,
            kind
     from fruit;
FILE_NUM BLOCK_NUM KIND
-------- --------- ------
       4       183 orange

SQL> select name from v$datafile where file# = 4;
NAME
-----------------------------
/u01/oradata/uw01/users01.dbf

Use the dd utility to skip to block 183 and extract the DB block containing the row:

unixhost% dd bs=8k skip=183 count=1 if=/u01/oradata/uw01/users01.dbf | strings
1+0 records in
16+0 records out
orange

Now we put the tablespace into hot backup mode:

SQL> alter tablespace users begin backup;
Tablespace altered.

Update the row, commit, and force a checkpoint on the database.

SQL> update fruit set kind = 'plum';
1 row updated

SQL> commit;
Commit complete.

SQL> alter system checkpoint;
System altered.

Extract the same block. It shows that the DB block has been written to disk during backup mode:

unixhost% dd bs=8k skip=183 count=1 if=/u01/oradata/uw01/users01.dbf | strings
1+0 records in
16+0 records out
plum
orange

Don’t forget to take the tablespace out of backup mode!

SQL> alter tablespace administrator end backup;
Tablespace altered.

It is quite clear from this demonstration that datafiles receive writes even during hot backup mode!

Related posts:

1. Exploring Oracle 11g Tablespace Encryption
2. Oracle Backup and Recovery
3. Emergency Database Recovery

At Blue Gecko, many of our remote DBA customers have been asking us about Golden Gate.  In July 2009, Oracle bought Golden Gate Software, just one of several companies that have developed log-based replication mechanisms for Oracle and other databases.  This was one of many major acquisitions by Oracle in 2009, including Sun and Relsys. But unlike most of Oracle’s acquisitions, Golden Gate provides very little new functionality not already available in Oracle Streams. Nevertheless, at OpenWorld 2009, Oracle made a shocking announcement.  They declared that Golden Gate would be the primary replication channel for Oracle, and that development would cease on Streams and related components.

Usually, Oracle watches these little third-party products for good ideas, then implements them independently (and better) on their own in the Oracle kernel, then watches as the little third-party companies fizzle out.  A case in point is direct memory access performance sampling.  Precise Software and several other companies in the early 2000s developed  low-impact performance sampling and visualization products for Oracle based on sampling the SGA periodically from an external program.  In version 10g, Oracle answered them with Active Session History (ASH), which did the same thing but better.  Although ASH required customers to purchase the Diagnostic Pack, it still more or less spelled the downfall of competing products.

But in the case of Golden Gate, Oracle already has a log-based replication technology (Streams) built into the kernel, available for a very reasonable price (free with Enterprise Edition).  The only major components that Streams lacks compared to Golden Gate is the ability to replicate across database platforms (Oracle to MSSQL, MySQL, etc. and vice versa).  Even that capability was clearly around the corner: In 11g, Logical Standby (Data Guard), a technology that uses essentially the same stack of components as Streams, gained cross-platform capabilities.

By 11g, Streams has become a mature and stable product, and is far more scalable and configurable than Golden Gate in many ways.  Streams can mine logs on the source or the target, or even a third system.  Depending on the load profile, you can use a wide variety of configuration choices, including parallelism at almost any point.  Streams also allows customers to choose to enforce transaction order or not.

In contrast, Golden Gate’s parallelism is restricted to the apply side, and in parallel mode, does not have the option of guaranteeing transaction order (it is non-ACID). Golden Gate’s parallel apply splits work up by schema, relying on the assumption that interdependent data at the business process level is confined to a single schema at a time.  In other words, if all the tables reside in one schema, then parallel apply doesn’t work, and if they reside in many schemas, the changes in one schema may be applied out of order vis à vis the changes to the other schemas.

Streams is only one of Oracle’s preexisting features that can compete successfully in specific use cases with Golden Gate.  Even more ancient and time-tested solutions such as advanced replication and remote materialized views remain supported and highly effective, depending on the requirement.

If you look at many of the use cases where our customers have deployed Golden Gate, I find that the simplest and most scalable engineering solution would have been remote fast-refresh materialized views.  Our customers often replicate core look-up data, like exchange rates, inventory levels, and other slowly-changing data between Oracle databases within an enterprise.  For this, Golden Gate is completely unjustified, due to cost and complexity compared to remote materialized views.  If it were a question of heterogeneous (inter database product) replication, I completely understand.  But in the majority of situations where we see Golden Gate in use, it is Oracle to Oracle. Given that, I wonder how it could come to pass that responsible people would recommend and implement a solution for such a requirement involving Golden Gate.  Why would Oracle essentially abandon ten years of development and stabilization on a platform like Streams for a less mature, rudimentary product like Golden Gate? Oracle can’t possibly be asking customers to pay additional license fees for a worse version of a product they already own.

So let’s review…

Streams: Mature, complex, requires engineering, highly configurable, scalable, Oracle-only, free.

Golden Gate: Simple, east to deploy, few configuration options, less scalable, expensive, heterogeneous (inter-RDBMS), might break your data.

For me, the corporate direction with regard to Golden Gate is perplexing and smacks of sales-driven (as opposed to requirements and cost-driven) engineering.  I can only imagine what it must be like for the team at Oracle that built Log Miner and AQ into an impressive suite of options including Streams.

UPDATE

Since I posted this, a colleague inside Oracle reassured me that although the product will bear the name ‘Golden Gate,’ it will incorporate features and capabilities of Streams and Golden Gate into a single suite of functions as versions are released over time.  This means that the declarations from inside Oracle do not represent the abandonment of ten years of development.

Related posts:

1. Oracle Education – Our First DBA Class!!
2. Blue Gecko named to 2010 Inc. Magazine’s 5000 fastest growing companies in America!
3. Report from Oracle Openworld

• OTN is free for anyone, and provides discussion forums, white papers and free base-release software downloads under the Developer License.
• MOS requires a paid software license and support contract. It offers technical support notes, bug access, request tracking and patches.  It is the only place you can get critical updates and patch sets to, for instance, bring a 10.2.0.1 database up to 10.2.0.4.

Many people are just starting learning Oracle, or are out of work and no longer associated with a support contract.  Those people could clearly benefit from access to MOS to read support notes, and solve problems they are having while experimenting with Oracle products under the Developer License.

A few years ago, my friend and Oracle ACE Director Dan Morgan mentioned that one could buy a very cheap product in the Oracle Store, opt to pay for support on it, and obtain MOS access for very little money.  This week I mentioned this fact to my Oracle DBA Class at the University of Washington, and my students wanted to know which product to buy, and for how much.  I decided to find out, and here’s what I did.

1. Navigate to http://shop.oracle.com.
2. In the left-hand menu, click on All Products at the bottom.
3. In the gray bar with heading labels, click on Price Range (on the right), and select $.40 – $8.00.
4. The only item that appears for me is Oracle Management Pack Plus for Identity Management. Click on Buy Now.
5. On the product page, change the Metric selection to Non Employee User External – $2.00.
6. Click Add to Cart.
7. Click Checkout (upper right)
8. Make sure you have checked Include First Year Support for License Products.
9. Click Checkout, and follow the purchase process.
10. In a day or so, you will receive an email with your Customer Service Identifier (CSI) and you will be able to register for My Oracle Support.

NOTE: It may take more than 24 hours for Oracle to activate your CSI on MOS. If you try to reigster your new CSI on MOS before Oracle activates it, you will receive the error, “This CSI does not allow registration.” In this case, just wait.  Oracle will activate your CSI eventually.

This process doesn’t give you the right to download or use material from the MOS site for any purpose not covered under the Developer License or the product license you purchased for $2.00 from the Oracle Store.  I am providing this information because it could help those legitimately covered under the Developer License.

Related posts:

1. Oracle Certification vs Oracle Support
2. Oracle (finally) announces support for Oracle Database 11gR2 on OEL6
3. Oracle Support and Certification on AWS