This page is a collection of DBA horror stories. It is intended to educate and entertain. The stories are true, and anonymous to encourage those who may be too embarrassed or otherwise consider it undignified to identify themselves or the companies involved. I've only received a couple so far, but hopefully they will encourage others to come forward - I'll anonymize them unless specifically instructed otherwise. At some point I will also have pages of my own stories (when I figure out how to do that without upsetting potential employers or customers!) and interesting stuff found while surfing (which of course anyone can compile - but who does?). This page is for first person horror experiences. So fess up, people! I know you're out there, 'cause we've all had to learn somehow! And "somehow" often means "screwed up bigtime!" We all know that Oracle doesn't have any bugs and DBA's are always right and never lie, eh? :-)
My own comments are in square brackets [].
========================================================================
I tried to backup a database once with "tar -cf". I carefully listed every needed file (redo logs, control files, datafiles) on the command line and pressed ENTER, only to realize seconds later that I'd forgotten to put the destination file first in the list. This backed up files 2 though n over top of file 1. After changing my underwear, I realized that file 1 was a mirrored redo log. If it had been a datafile I'dve been up the creek.
How's this compare with some of the others?
[A good scare, at least! :) A good idea for an award if this ever gets to be an IOUG presentation - perhaps "The Ruined Underwear"? Or maybe "The Runed Underwear" for mistakes due to arcane commands! ]
I also once had a nightmare that someone accidentally switched the names of my production and test SQLNet service names. But since that never actually happened I guess it doesn't count.
[It's happened... ;-0 Anything can happen when non-dba's start tinkering with these .ora things, I actually had this happen in two ways at the same time - an engineer didn't understand the syntax, so simply duped the entire section defining node names. This has the effect of only using the last definition, and silly me was going nutz trying to figure out what was wrong, not scrolling down from the first definition which I had modified correctly after he had copied it lower down. Similarly, the (otherwise capable) unix admin had decided to switch a couple of nodenames and addresses, which /etc/hosts of course didn't get propagated to all machines...]
===================================================================================
I lasted 2 days at the ORACLE DBA consulting position described below:
- all developers has DBA priviliges and would constantly change tablespace/table/column names
- the application had NO documentation and NOBODY had the true source code of what was actually running.
- this app had the machine names, schemas, tablespaces and tables HARDCODED into it. Hence when somebody changed anything... the application would break.
- they INSISTED that rule-based optimization was the THE WAY, yet their SQL was not tuned for any optimization.
- their tuning methodology was to add more memory to the server.
- the app would run via some tables in the production instance, some tables in the development instance, and even some local files residing in some temporay employee's user directory. Evidently this temp employee was some grad student in computer science who had zero experience but they worshiped ever utterance from him.
- backups were performed by making unix tar files of "some" of the database and application files ( nobody truly knew all of the files that were being used ); but the database was not taken down or running in hot backup mode.
- if it weren't for all of the above, NOBODY was ever allowed to take the database down for any reason. Not even for a cold backup....
I told them they had a house-of-cards of a database and to make sure they didn't forget how to work the deep fryer in Kentucky Fried Chicken because that would be their next career move.
After I stopped shaking my head at them, I started laughing and told them that when it crashed, not ever to call me or my agency for help.
[This one truly frightened me, as I thought it was describing the job I had just taken! Then I realized it couldn't be, they didn't know how to have more than one instance. However, I've stuck it out, it's been quite entertaining. Basically six months worth of work to straighten most of it out. One persons horror may be anothers opportunity, eh? Getting all that stuff out of the system tablespace without anyone noticing made for a bit of nailbiting, I'll tell ya'.]
===================================================================================
Hi Joel,
I'm a programmer in a local company and have to deal with Oracle,too.
You won't believe what just happened to our DB:
99% of the INSERT-triggers and other SPs went crazy. SELECT from DUALs returned multiple rows =) - we inspected SYS.DUAL and found some strange entries there. When I checked the open sessions I found the statement"select * from dual where dummy <> 'X' order by dummy;"
executed by a co-worker. I think you'll guess by now what happened:
He found your posting
"Joel Garry"
> jg
> --
> insert into dual values ('N');
> insert into dual values ('M');
> insert into dual values ('A');
> select * from dual where dummy <> 'X' order by dummy;
but didn't understand it at all and wanted to try "what happens when I execute this." (he expected some magic). [We're in development here and everyone has the SYS-Login.]
Now he's got one less testicle. I think he'll be fine by next week ;-)
Regards & thanks for the demonstration of human ignorance,
===================================================================================
===================================================================================
Thread on SQL Server v. Oracle
===================================================================================
Oracle-l thread on database oops
===================================================================================
===================================================================================