{"id":1072,"date":"2006-08-31T12:00:00","date_gmt":"2006-08-31T12:00:00","guid":{"rendered":"http:\/\/orcldoug.com\/blog\/?p=1072"},"modified":"2006-08-31T12:00:00","modified_gmt":"2006-08-31T12:00:00","slug":"if-youre-gonna-screw-up","status":"publish","type":"post","link":"http:\/\/orcldoug.com\/blog\/2006\/08\/31\/if-youre-gonna-screw-up\/","title":{"rendered":"If you&#8217;re gonna screw up &#8230;"},"content":{"rendered":"<p>&#8230; you should always aim <strong>high<\/strong> (or desperately low would be more accurate). <\/p>\n<p>I was sitting at my desk today, deleting some old databases (that phrase in itself should be warning enough) when I typed in the following<\/p>\n<pre><code>dunha1e:oracle#rm -fR *<\/code><\/pre>\n<p>I looked down at my notes to have a think about what I was doing, then typed this<\/p>\n<pre><code>ls -ltra \/ora\/data2<\/code><\/pre>\n<\/p>\n<p>What might not be so obvious from the way I&#8217;ve written that out is that I was missing a CR\/LF in there, so what I <em>actually<\/em> typed was<\/p>\n<pre><code>dunha1e:oracle#rm -fR *ls -ltra \/ora\/data2<\/code><\/pre>\n<p>[dripping_sarcasm]Fortunately I managed to remember the CR\/LF after the &#8216;second command&#8217;.[\/dripping_sarcasm]<\/p>\n<p>I realised what I&#8217;d done <i>immediately<\/i>. For those who aren&#8217;t rolling about on the floor laughing or absolutely disgusted already, I deleted the following<\/p>\n<ul>\n<li>*ls<\/li>\n<li>-ltra<\/li>\n<li>\/ora\/data2 and all of it&#8217;s subdirectories and all of the contents<\/li>\n<\/ul>\n<p>First, the good news (there&#8217;s not much)<\/p>\n<ul>\n<li>This was a development server. That&#8217;s probably why I was more casual than normal but, make no mistake, if you&#8217;re a developer or a tester &#8211; this <i>is<\/i> Production for you.<\/li>\n<li>We use RMAN so recovery should be straightforward.<\/li>\n<li>In spite of their baser instincts, everyone rallied round to the cause and my team lead didn&#8217;t have a fit<\/li>\n<\/ul>\n<p>Now some bad<\/p>\n<ul>\n<li>There were about eight databases using files on \/ora\/data2!<\/li>\n<li>The thing with development servers is that they&#8217;re more of an untidy mess with less surrounding structure and procedures. Which meant a lot more thought needed to be put into what steps would get us back to where we were. (Entertainingly, I&#8217;d had a strong argument with my team lead earlier in the day about imposing more structure on things. Maybe he&#8217;ll do that by revoking my access privileges &#128521;)<\/li>\n<li>For example, we don&#8217;t back up all of the databases on that server. For some of them we rely on exports. I happen to think an export should never be considered a <i>backup<\/i> of a database, as we&#8217;re about to find out when we try to recover them &#128577;<\/li>\n<li>When I started to recover the first few databases, our standard recovery shell-script was falling over. (No recovery test, you see)<\/li>\n<li>I&#8217;ve only been on this new team a few weeks, so I don&#8217;t know the environment that well. Nothing beats practice and prior knowledge.<\/li>\n<li>When I started to use RMAN manually, I ran into <a href=\"http:\/\/tinyurl.com\/ovwmu\">the most bizarre bug<\/a> when I retried a recovery in the same session after one had failed. (You&#8217;ll need a Metalink account for that link.) We spent a bit of time sorting that one out!<\/li>\n<\/ul>\n<p>With the help of a couple of colleagues, we got the most important 3 databases recovered with no data loss and some flat-files restored too. However, there are about 5 databases that we&#8217;re going to need to recreate from scratch and then import export files. Normally I would have stayed, but I have a big meeting tomorrow I need to prepare for, so our evening shift guy is progressing the work and I&#8217;ll be in first thing in the morning to try to finish the clean up.<\/p>\n<p>In fairness, it&#8217;s a long time since I&#8217;ve made such a stupid mistake but there was some joking discussion about whether this would make it into my blog. Of course it would. Everyone makes stupid mistakes sometimes. However, a little more care would have avoided this (as would have the -i flag on the rm command!) and it was <i>particularly<\/i> stupid, so I&#8217;m not proud of it. However, what&#8217;s most important when you do make mistakes is<\/p>\n<ul>\n<li>You&#8217;re honest about them.<\/li>\n<li>You fix them.<\/li>\n<li>You learn from them.<\/li>\n<li>You thank anyone else who helped you fix them.<\/li>\n<\/ul>\n<p>I&#8217;ll get my coat &#128577;<\/p>\n<p><i>P.S. How&#8217;s this for a punchline? At the Miracle Database Forum in a few weeks time I&#8217;ll be giving a presentation in the subject area of DBA Worst Practices. Well, I reckon I probably have some suitable material!<\/i><\/p>\n","protected":false},"excerpt":{"rendered":"<p>&#8230; you should always aim high (or desperately low would be more accurate). I was sitting at my desk today, deleting some old databases (that phrase in itself should be warning enough) when I typed in the following dunha1e:oracle#rm -fR * I looked down at my notes to have a think about what I was&hellip; <a class=\"more-link\" href=\"http:\/\/orcldoug.com\/blog\/2006\/08\/31\/if-youre-gonna-screw-up\/\">Continue reading <span class=\"screen-reader-text\">If you&#8217;re gonna screw up &#8230;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-1072","post","type-post","status-publish","format-standard","hentry","category-uncategorized","entry"],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":1044,"url":"http:\/\/orcldoug.com\/blog\/2006\/08\/04\/tracing-session-activity-over-a-remote-database-link\/","url_meta":{"origin":1072,"position":0},"title":"Tracing session activity over a remote database link","date":"August 4, 2006","format":false,"excerpt":"Yesterday someone asked me how to trace a session that selects from a view in a remote database via a link. If they activated the trace on the local instance, they wouldn't see the bulk of the work which was happening on the remote instance - just a bunch of\u2026","rel":"","context":"With 2 comments","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1112,"url":"http:\/\/orcldoug.com\/blog\/2009\/10\/29\/10g-consolidated-trace-files-and-px\/","url_meta":{"origin":1072,"position":1},"title":"10g Consolidated Trace Files and PX","date":"October 29, 2009","format":false,"excerpt":"At the end of my Tracing Parallel Execution presentation at the Scottish OUG conference, Michael M\u00f8ller of Miracle asked whether the consolidated trace files produced by the DBMS_MONITOR package and trcsess utility are in strict time order. i.e. Will I see the actions of the query coordinator, then the actions\u2026","rel":"","context":"With 7 comments","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1063,"url":"http:\/\/orcldoug.com\/blog\/2006\/08\/21\/10g-default-installation-and-ofa\/","url_meta":{"origin":1072,"position":2},"title":"10g Default Installation and OFA","date":"August 21, 2006","format":false,"excerpt":"or ... 'what's that all about?!?!'Last year, I blogged about some of the variations on Optimal Flexible Architecture that I've seen on my travels. I'm gradually coming round to having the service group for clustered applications as the top directory level (e.g.\/myapp1\/oradata\/\/myapp1\/oraindex\/\/myapp2\/oradata\/\/myapp2\/oraindex\/but I'm still not convinced. If people ever failed\u2026","rel":"","context":"With 7 comments","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1264,"url":"http:\/\/orcldoug.com\/blog\/2007\/05\/06\/re-linking-oracle\/","url_meta":{"origin":1072,"position":3},"title":"Re-linking oracle","date":"May 6, 2007","format":false,"excerpt":"Hooray! Something vaguely technical - at last.My new client is hot on software licence compliance and what seemed a minor issue to me reared it's slightly ugly head this week. When connecting to our 10.2.0.2 database instances using sqlplus, the version banner displays options that we aren't using, for example\u2026","rel":"","context":"With 21 comments","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1272,"url":"http:\/\/orcldoug.com\/blog\/2007\/05\/24\/listener-bug-part-2\/","url_meta":{"origin":1072,"position":4},"title":"Listener bug &#8211; Part 2","date":"May 24, 2007","format":false,"excerpt":"Work is a little busy at the moment, but I had time to look at this a bit more today ... Many thanks for the useful follow-up comments on the last posting. They were very interesting and if I'd picked them up in some of my Google searches, we might\u2026","rel":"","context":"Similar post","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1320,"url":"http:\/\/orcldoug.com\/blog\/2007\/09\/23\/parallel-query-and-11g-part-2\/","url_meta":{"origin":1072,"position":5},"title":"Parallel Query and 11g &#8211; Part 2","date":"September 23, 2007","format":false,"excerpt":"Now, that's weird. A little surprising might be more accurate and maybe I'm missing something.During the various tests with and without parallel hints and different parallel_io_cap_enabled settings, I expected the runs that didn't use parallelism to show up \"db file scattered read\" events in the trace files. For example, here's\u2026","rel":"","context":"With 10 comments","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"http:\/\/orcldoug.com\/blog\/wp-json\/wp\/v2\/posts\/1072","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/orcldoug.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/orcldoug.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/orcldoug.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/orcldoug.com\/blog\/wp-json\/wp\/v2\/comments?post=1072"}],"version-history":[{"count":0,"href":"http:\/\/orcldoug.com\/blog\/wp-json\/wp\/v2\/posts\/1072\/revisions"}],"wp:attachment":[{"href":"http:\/\/orcldoug.com\/blog\/wp-json\/wp\/v2\/media?parent=1072"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/orcldoug.com\/blog\/wp-json\/wp\/v2\/categories?post=1072"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/orcldoug.com\/blog\/wp-json\/wp\/v2\/tags?post=1072"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}