Ask The Real Tom

May 28, 2008



Row Migration and Row Chaining - Your Developer knows best!

Filed under: Uncategorized — Tags: , , — admin @ 5:24 pm

So today a developer called me an said the production db is slow in doing some updates and the reason is row chaining or row migration. On the development db he does 60 updates a second, on production only about 2 or 3.

Did I tell you already that he is a developer? :-)

Let me explain the situation

  • 6′500′000 rows
  • two attributes defined as varchar(2000)
  • one of the varchar get fille at insert time, the other later with an update
  • characterset AL32UTF8 (1-4 bytes)
  • varchar(x) means x bytes and not x characters
  • max record size about 2k + 2k + 100bytes = ~4k
  • block size 8k

The situation look like row migration could happen often. I startet to check it. First let’s check the “fetch continued row”. This gives an evidence of row chaining or row migration. It indicated how many time since the last startup (2 days ago) a second block hat to be loaded.
SELECT name, value FROM v$sysstat WHERE name = 'table fetch continued row';
NAME VALUE
---------------------------------------------------------------- ----------
table fetch continued row 706806

Well, the is something. Let’s check deeper. The updates happen on the table BUUBUU (anoymized table name for security reason, the application is high confidential).

Standard Check
SQL> @utlchain.sql -- installed the the chained_rows table
SQL> ANALYZE TABLE BUUBUU LIST CHAINED ROWS;
-- LIST CHAINED ROWS lets you identify migrated and chained rows of the analyzed table.
SQL> ANALYZE TABLE scott.emp LIST CHAINED ROWS;
SQL> SELECT * FROM chained_rows

Thath is about 1.3% of the records. No big deal.
But let’s check more. How many rows are in a block. First I tried to crap around with the vsize stuff found in the internet. After 30 min. needed one of my davidoff mini cigarillos - while smoking I hat an idea. Why use the inflexible vsize stuff to get a imprecise result. The ROWID will me tell an excat result.

Detailed Result for each block:
select substr(rowid,1,6) "object",
substr(rowid,7,3) "file",
substr(rowid,10,6) "block",
count(substr(rowid,16,3)) "countrows"
from BUUBUU where rownum < 30
group by substr(rowid,1,6), substr(rowid,7,3), substr(rowid,10,6)
order by count(substr(rowid,16,3))
/

Summary:
select count(1) ,sub.countrows from (
select substr(rowid,1,6) "object",
substr(rowid,7,3) "file",
substr(rowid,10,6) "block",
count(substr(rowid,16,3)) countrows
from BUUBUU
group by substr(rowid,1,6), substr(rowid,7,3), substr(rowid,10,6)
) sub
group by sub.countrows
order by 1
/

The result:
COUNT(1) COUNTROWS
---------- ----------
89 12
1437 4
1876 3
2056 2
2577 1
2599 10
2684 5
3273 9
4940 8
25696 11
432275 6
600786 7

12 rows selected.
We see most blocks contains from 6 to 7 rows. (let’s discuss about a bigger block size another time).

Futher I wanted to check about the update statments. How many rows are updated with the pricemessage.
select count(1) from BUUBUU where BLOBOBJECT is null;
COUNT(1)
----------
0
select count(1) from BUUBUU where PRICEMESSAGE is null;
COUNT(1)
----------
6911440

select count(1) from BUUBUU;
COUNT(1)
----------
7238457

Just 327017 record got an update. That’s aobut (ca. 5%)

My position is now clear, that this not a real issue on this table (may be a point for optimization).
Tomorrow we will do a trace the seek the wait times (I could tell you the result already, it will be net waites)

Btw. If you have a problem with row migration or row chaining:

  • Row Chaining: Make your blocks bigger or use LOB’s not stored inline.
  • If you have row migration think about increasing your PCTFREE on the table

May 27, 2008



Oracle Patch 10.2.0.4 for Sun Solaris is out now!

Filed under: oracle — Tags: , , , — admin @ 7:19 am

Now the oracle patch 10.2.0.4 for solaris is out.

This patch includes all CPU patchs including the “April 2008 Critical Patch Update”

The only issue known know is something on windows (but we don’t care since windows is not an operating system, it’s more something like BIOS extension).

And there is a relinking problem. If you do a

relink all

you get

make: Fatal error: Don’t know how to make target 'install'

But it’s easy to solve. Just rename the $ORACLE_HOME/odbc/lib/ins_odbc.mk file tometing else.

May 15, 2008



Solaris error 12, not enough space

Filed under: operating system — Tags: , , , , , — admin @ 11:37 pm

Who don’t know the Solaris errors:

  • unable to fork a process
  • Solaris error 12, not enough space
  • TNS-12500, TNS-12540, TNS-12560, TNS-00510

This errors are memory or swap related. Spezially swap allocation seems to be a mystery under solaris.
A description about solaris system errors is avaliable with the command:

man -s2 intro

So the question is: How much swap space do I need on solaris to run a oracle database?

An extract from somewhere describing solaris swap allocation:

The Virtual Swap Reserved is 12 gigabytes. This is the same as the application memory
size shown in the prtmem output. This is because for each application page a
corresponding backing storage is reserved on disk even though it is not being used. This
storage is just in case the system reaches critical shortage and the page has to be copied
out to disk during a write.

So if don’t want to deal with this error, just allocate at leas as much swap space as you have physical memory. Even your system administrator tell you something else (he don’t know).

May 14, 2008



Multi CPU bzip2

Filed under: operating system — Tags: , , , , — admin @ 9:08 pm

Not really oracle related, but a interesting thing and it’s performance related (which is my nr.1 passion)

Haven’t you been upset that bzip2 takes sooo long? Is CPU intensive but just uses 1 CPU (or core).

The solution is pbzip2 which can use multiple cpu’s

My Teste:

CPU: ntel(R) Core(TM)2 CPU 4400 @ 2.00GHz (cache: 2048 KB)

FIle: (mysql dump)

562M 2008-05-14 21:30 shopprod200805132200.dmp

bzip2 run

time bzip2 shopprod200805132200.dmp

real 3m23.115s
user 3m14.252s
sys 0m2.632s

time bunzip2 shopprod200805132200.dmp.bz2

real 1m43.184s
user 1m38.118s
sys 0m2.748s

pbzip2 run

time pbzip2 shopprod200805132200.dmp
real 2m29.084s
user 4m0.855s
sys 0m8.561s
time pbzip2 -d shopprod200805132200.dmp.bz2

real 0m59.406s
user 1m46.087s
sys 0m5.744s

So on my machine compression is about 36% faster. Decompression about 76 % faster.

May 4, 2008



Tune a oracle database for performance (II)

Filed under: oracle — Tags: , , , , , — admin @ 11:26 pm

It’s about time to give you an update on my tuning challange.

Few statments:

  • bad perfomance is better than unstable or changing performance
  • if you gather system stats to a stats table the intervall option does not work (I did it with cronjob to start and stop the gathering)

My old system stats looked like this:

col PVAL2 for a50
select * from sys.aux_stats$;
SNAME                          PNAME                               PVAL1 PVAL2
------------------------------ ------------------------------ ---------- -------------
SYSSTATS_INFO                  STATUS                                    COMPLETED
SYSSTATS_INFO                  DSTART                                    10-13-2006 05:41
SYSSTATS_INFO                  DSTOP                                     10-13-2006 05:41
SYSSTATS_INFO                  FLAGS                                   1
SYSSTATS_MAIN                  SREADTIM                              1.8
SYSSTATS_MAIN                  MREADTIM                              1.3
SYSSTATS_MAIN                  CPUSPEED                              590
SYSSTATS_MAIN                  MBRC                                    8
SYSSTATS_MAIN                  MAXTHR                          377000000
SYSSTATS_MAIN                  SLAVETHR                           131000

The new ones gathered from production and ajusted a bit like this.

exec DBMS_STATS.SET_SYSTEM_STATS('mbrc',12);
exec DBMS_STATS.SET_SYSTEM_STATS('sreadtim',1.8);
exec DBMS_STATS.SET_SYSTEM_STATS('mreadtim',2.2);
exec DBMS_STATS.SET_SYSTEM_STATS('cpuspeednw',500);
exec DBMS_STATS.SET_SYSTEM_STATS('ioseektim',12);
exec DBMS_STATS.SET_SYSTEM_STATS('iotfrspeed',19000);

The gathered mbrc was just 2. But that value was gathered over 7 hours an may not represent the peak load. I ajusted it to 12 just because I feel good with 12. (42 might be the best value, but lets stick with 12)

The mreadtim was lower as the sreadtim because of our multi-oi-optimized SAN. But the workload statistiks don’t get used if mreatime < sreadtime. I read somewhere (don’t rember where) that sreadtim*1.2 might be a reasonable value.

The CPU speed is set lower now, because the CPU workload on the system increased much.

If my new system stats do good… We will see after the first integration test.

The second tuning step will be sql profiles. It’s too dangerous to implement them on production. So I decided to do remote tuning.

  • Fist get a SQL Tuning Set (STS) on production
  • Load it to a staging table and export it
  • Import the staging table on the integration environment and unpack the STS
  • Let dbms_sqltune do some analysis on it
  • Implement the sql profiles an test their effect
  • If they are good (one is for sure 16s -> 0.01s) load them to a staging table
  • Import the sql profiles on the Production and unpack them

Some sql used to perform this step:

#create the sts staging table
BEGIN
  DBMS_SQLTUNE.CREATE_STGTAB_SQLSET( table_name => 'staging_table' );
END;
/ 

# put my sts to the staging table
BEGIN
  DBMS_SQLTUNE.PACK_STGTAB_SQLSET(
      sqlset_name  => 'my_sts',
      staging_table_name => 'staging_table');
END;
/ 

# create sql profile staging table
exec DBMS_SQLTUNE.CREATE_STGTAB_SQLPROF ( table_name => 'SQLPROF_STAGING', schema_name => 'SYSTEM' );

# move all my sql profiles to the staging table
begin
DBMS_SQLTUNE.PACK_STGTAB_SQLPROF (
   profile_name          => '%',
   profile_category      => 'DEFAULT',
   staging_table_name    => 'SQLPROF_STAGING',
   staging_schema_owner  => 'SYSTEM');
end;
/ 

# unpack all (%) my sql profiles on production
begin
DBMS_SQLTUNE.UNPACK_STGTAB_SQLPROF (
   profile_name          => '%',
   profile_category      => 'DEFAULT',
   replace               => true,
   staging_table_name    => 'SQLPROF_STAGING',
   staging_schema_owner  => 'SYSTEM');
end;
/

Please post a comment if you think there is a better way.

Powered by WordPress