How to delete the “site” in your asm disks without recreating theirs diskgroups
(First of all, I would like to thank the help from my BIG friend Óscar, he created the script with awk for doing the change)
In a Grid Infrastructure 12.2 version, we have configured the AFD (asm filter driver) and some diskgroups created with normal redundancy. They belong, by mistake, to some sites, it was configured a extended cluster but they are not using them but the voting disks and the ocr.
On these diskgroups are created some volumes and on the top of these, some filesystems, obviously if you want to change this information, you’d have to recreate everything from scratch for creating a new diskgroups without the “site” definition.
So we are going to try it with less work 😉
You can pay attention to the column “site_name”:
SYS@+ASM1> col path format a20
SYS@+ASM1> col failgroup format a20
SYS@+ASM1> col name format a20
SYS@+ASM1> col site_name format a20
SYS@+ASM1> select INST_ID,GROUP_NUMBER,DISK_NUMBER,MOUNT_STATUS,HEADER_STATUS,MODE_STATUS,STATE,NAME,FAILGROUP,SITE_NAME,PATH from gv$asm_disk order by 1,2,3;
INST_ID GROUP_NUMBER DISK_NUMBER MOUNT_S HEADER_STATU MODE_ST STATE NAME FAILGROUP SITE_NAME PATH
---------- ------------ ----------- ------- ------------ ------- -------- -------------------- -------------------- --------- --------------------
[...]
1 3 0 CACHED MEMBER ONLINE NORMAL PROMGMTDB101 FGPROMGMTDB1 site1 AFD:PROMGMTDB101
1 3 1 CACHED MEMBER ONLINE NORMAL PROMGMTDB202 FGPROMGMTDB2 site2 AFD:PROMGMTDB202
1 3 2 CACHED MEMBER ONLINE NORMAL PROMGMTDB201 FGPROMGMTDB2 site2 AFD:PROMGMTDB201
1 3 3 CACHED MEMBER ONLINE NORMAL PROMGMTDB102 FGPROMGMTDB1 site1 AFD:PROMGMTDB102
[...]
1 6 0 CACHED MEMBER ONLINE NORMAL PRODATA101 FGPRODATA1 site1 AFD:PRODATA101
1 6 1 CACHED MEMBER ONLINE NORMAL PRODATA202 FGPRODATA2 site2 AFD:PRODATA202
1 6 2 CACHED MEMBER ONLINE NORMAL PRODATA103 FGPRODATA1 site1 AFD:PRODATA103
1 6 3 CACHED MEMBER ONLINE NORMAL PRODATA104 FGPRODATA1 site1 AFD:PRODATA104
1 6 4 CACHED MEMBER ONLINE NORMAL PRODATA105 FGPRODATA1 site1 AFD:PRODATA105
1 6 5 CACHED MEMBER ONLINE NORMAL PRODATA106 FGPRODATA1 site1 AFD:PRODATA106
1 6 6 CACHED MEMBER ONLINE NORMAL PRODATA203 FGPRODATA2 site2 AFD:PRODATA203
1 6 7 CACHED MEMBER ONLINE NORMAL PRODATA204 FGPRODATA2 site2 AFD:PRODATA204
1 6 8 CACHED MEMBER ONLINE NORMAL PRODATA201 FGPRODATA2 site2 AFD:PRODATA201
1 6 9 CACHED MEMBER ONLINE NORMAL PRODATA107 FGPRODATA1 site1 AFD:PRODATA107
1 6 10 CACHED MEMBER ONLINE NORMAL PRODATA102 FGPRODATA1 site1 AFD:PRODATA102
1 6 11 CACHED MEMBER ONLINE NORMAL PRODATA108 FGPRODATA1 site1 AFD:PRODATA108
1 6 12 CACHED MEMBER ONLINE NORMAL PRODATA109 FGPRODATA1 site1 AFD:PRODATA109
1 6 13 CACHED MEMBER ONLINE NORMAL PRODATA205 FGPRODATA2 site2 AFD:PRODATA205
1 6 14 CACHED MEMBER ONLINE NORMAL PRODATA206 FGPRODATA2 site2 AFD:PRODATA206
1 6 15 CACHED MEMBER ONLINE NORMAL PRODATA207 FGPRODATA2 site2 AFD:PRODATA207
1 6 16 CACHED MEMBER ONLINE NORMAL PRODATA208 FGPRODATA2 site2 AFD:PRODATA208
1 6 17 CACHED MEMBER ONLINE NORMAL PRODATA209 FGPRODATA2 site2 AFD:PRODATA209
1 7 0 CACHED MEMBER ONLINE NORMAL PROFRA101 FGPROFRA1 site1 AFD:PROFRA101
1 7 1 CACHED MEMBER ONLINE NORMAL PROFRA202 FGPROFRA2 site2 AFD:PROFRA202
1 7 2 CACHED MEMBER ONLINE NORMAL PROFRA103 FGPROFRA1 site1 AFD:PROFRA103
1 7 3 CACHED MEMBER ONLINE NORMAL PROFRA104 FGPROFRA1 site1 AFD:PROFRA104
1 7 4 CACHED MEMBER ONLINE NORMAL PROFRA105 FGPROFRA1 site1 AFD:PROFRA105
1 7 5 CACHED MEMBER ONLINE NORMAL PROFRA106 FGPROFRA1 site1 AFD:PROFRA106
1 7 6 CACHED MEMBER ONLINE NORMAL PROFRA201 FGPROFRA2 site2 AFD:PROFRA201
1 7 7 CACHED MEMBER ONLINE NORMAL PROFRA107 FGPROFRA1 site1 AFD:PROFRA107
1 7 8 CACHED MEMBER ONLINE NORMAL PROFRA203 FGPROFRA2 site2 AFD:PROFRA203
1 7 9 CACHED MEMBER ONLINE NORMAL PROFRA108 FGPROFRA1 site1 AFD:PROFRA108
1 7 10 CACHED MEMBER ONLINE NORMAL PROFRA109 FGPROFRA1 site1 AFD:PROFRA109
1 7 11 CACHED MEMBER ONLINE NORMAL PROFRA110 FGPROFRA1 site1 AFD:PROFRA110
1 7 12 CACHED MEMBER ONLINE NORMAL PROFRA102 FGPROFRA1 site1 AFD:PROFRA102
1 7 13 CACHED MEMBER ONLINE NORMAL PROFRA204 FGPROFRA2 site2 AFD:PROFRA204
1 7 14 CACHED MEMBER ONLINE NORMAL PROFRA205 FGPROFRA2 site2 AFD:PROFRA205
1 7 15 CACHED MEMBER ONLINE NORMAL PROFRA206 FGPROFRA2 site2 AFD:PROFRA206
1 7 16 CACHED MEMBER ONLINE NORMAL PROFRA207 FGPROFRA2 site2 AFD:PROFRA207
1 7 17 CACHED MEMBER ONLINE NORMAL PROFRA208 FGPROFRA2 site2 AFD:PROFRA208
1 7 18 CACHED MEMBER ONLINE NORMAL PROFRA209 FGPROFRA2 site2 AFD:PROFRA209
1 7 19 CACHED MEMBER ONLINE NORMAL PROFRA210 FGPROFRA2 site2 AFD:PROFRA210
1 8 0 CACHED MEMBER ONLINE NORMAL VOTOCR101 FGVOTOCR101 site1 AFD:VOTOCR101
1 8 1 CACHED MEMBER ONLINE NORMAL VOTOCR102 FGVOTOCR102 site1 AFD:VOTOCR102
1 8 2 CACHED MEMBER ONLINE NORMAL VOTOCR103 FGVOTOCR103 site1 AFD:VOTOCR103
1 8 3 CACHED MEMBER ONLINE NORMAL VOTOCR201 FGVOTOCR201 site2 AFD:VOTOCR201
1 8 4 CACHED MEMBER ONLINE NORMAL VOTOCR202 FGVOTOCR202 site2 AFD:VOTOCR202
1 8 5 CACHED MEMBER ONLINE NORMAL VOTOCR203 FGVOTOCR203 site2 AFD:VOTOCR203
1 8 6 CACHED MEMBER ONLINE NORMAL VOTOCR301 FGVOTOCR301 site3 AFD:VOTOCR301
83 rows selected.
As we are deleting this information, not changing it or adding it, we are not using the kfed’s option:
si/telbl KFED Site Label to stamp [sitelbl=string]
We create a directory where dump the files with the metadata information (orig), and another directory where we modify them (mod), so we have a backup for reversing the changes to their original data in case of problems 😉
[root@pepito1 ~]# mkdir -p /home/grid/arumel/kfed/orig [root@pepito1 ~]# mkdir -p /home/grid/arumel/kfed/mod
We create the script which generate the dumps, except for the diskgroups with the ocr and voting disks (in our case the 8) because these have to have them configured (in our case):
spool /home/grid/arumel/kfed/genera_dmps.sh select 'echo "Generating the file /home/grid/arumel/kfed/orig/' || name || '.dump ................" kfed op=read dev=/dev/mapper/' || name || 'p1 text=/home/grid/arumel/kfed/orig/' || name || '.dump chksum=yes verb=yes' from v$asm_disk where site_name is not null and group_number!=8 order by name; spool off
The output of this script will be something like this (putting the output in the directory “orig”):
[...] echo "Generating the file /home/grid/arumel/kfed/orig/PRODATA101.dump ................" kfed op=read dev=/dev/mapper/PRODATA101p1 text=/home/grid/arumel/kfed/orig/PRODATA101.dump chksum=yes verb=yes [...]
We create the script which upload the modified data in the disks again (important to notice that the directory is “mod”!!)
spool /home/grid/arumel/kfed/carga_dmps.sh select 'echo "Uploading the file /home/grid/arumel/kfed/mod/' || name || '.dump ................" kfed op=write dev=/dev/mapper/' || name || 'p1 text=/home/grid/arumel/kfed/mod/' || name || '.dump chksum=yes verb=yes' from v$asm_disk where site_name is not null and group_number!=8 order by name; spool off
The output from this script will create some lines like these:
[...] echo "Uploading the file /home/grid/arumel/kfed/mod/PRODATA101.dump ................" kfed op=write dev=/dev/mapper/PRODATA101p1 text=/home/grid/arumel/kfed/mod/PRODATA101.dump chksum=yes verb=yes [...]
We check the newly created files:
[root@pepito1 ~]# cd /home/grid/arumel/kfed/ [root@pepito1 kfed]# ls carga_dmps.sh genera_dmps.sh mod orig
We run as ‘root’ the script which will generate the dump files with the information inside of our asm disks (with the kfed operation ‘read’ you don’t need to stop the afd):
[root@ctbl109 kfed]# bash genera_dmps.sh Generating the file /home/grid/arumel/kfed/orig/PRODATA101.dump ................ kfedDiscover: Discovering device kfedDiscover: /dev/mapper/PRODATA101p1 is a UFS disk kfedDevOpen: Opening device /dev/mapper/PRODATA101p1 from previous discovery NOTE: kfedOperation : op=read dev=/dev/mapper/PRODATA101p1 text=/home/grid/arumel/kfed/orig/PRODATA101.dump chksum=yes verb=yes (RESULT : Success) Generating the file /home/grid/arumel/kfed/orig/PRODATA102.dump ................ kfedDiscover: Discovering device kfedDiscover: /dev/mapper/PRODATA102p1 is a UFS disk kfedDevOpen: Opening device /dev/mapper/PRODATA102p1 from previous discovery NOTE: kfedOperation : op=read dev=/dev/mapper/PRODATA102p1 text=/home/grid/arumel/kfed/orig/PRODATA102.dump chksum=yes verb=yes (RESULT : Success) [...]
If you compare the output from a disk without the “site” configured with one that does have it, you can realize the difference between the last two numbers in each line which have to coincide with the siteGUID from the command “crsctl query cluster site -all” (in our case”37c347fd2c83ffb9bf127d0aca787ca9″):
Site 'site1' identified by GUID '37c347fd2c83ffb9bf127d0aca787ca9' in state 'ENABLED' [...]
0x37 0xc3 0x47 0xfd 0x2c 0x83 0xff 0xb9 0xbf 0x12 0x7d 0x0a 0xca 0x78 0x7c 0xa9
[root@pepito1 kfed]# grep siteguid prueba.dmp kfdhdb.siteguid[0]: 55 ; 0x088: 0x37 kfdhdb.siteguid[1]: 195 ; 0x089: 0xc3 kfdhdb.siteguid[2]: 71 ; 0x08a: 0x47 kfdhdb.siteguid[3]: 253 ; 0x08b: 0xfd kfdhdb.siteguid[4]: 44 ; 0x08c: 0x2c kfdhdb.siteguid[5]: 131 ; 0x08d: 0x83 kfdhdb.siteguid[6]: 255 ; 0x08e: 0xff kfdhdb.siteguid[7]: 185 ; 0x08f: 0xb9 kfdhdb.siteguid[8]: 191 ; 0x090: 0xbf kfdhdb.siteguid[9]: 18 ; 0x091: 0x12 kfdhdb.siteguid[10]: 125 ; 0x092: 0x7d kfdhdb.siteguid[11]: 10 ; 0x093: 0x0a kfdhdb.siteguid[12]: 202 ; 0x094: 0xca kfdhdb.siteguid[13]: 120 ; 0x095: 0x78 kfdhdb.siteguid[14]: 124 ; 0x096: 0x7c kfdhdb.siteguid[15]: 169 ; 0x097: 0xa9
You can pay attention at the column before the “;” which it would be “0” if you compare it with another disk without the “site” configured:
[root@pepito1 kfed]# grep siteguid prueba2.dmp kfdhdb.siteguid[0]: 0 ; 0x088: 0x00 kfdhdb.siteguid[1]: 0 ; 0x089: 0x00 kfdhdb.siteguid[2]: 0 ; 0x08a: 0x00 kfdhdb.siteguid[3]: 0 ; 0x08b: 0x00 kfdhdb.siteguid[4]: 0 ; 0x08c: 0x00 kfdhdb.siteguid[5]: 0 ; 0x08d: 0x00 kfdhdb.siteguid[6]: 0 ; 0x08e: 0x00 kfdhdb.siteguid[7]: 0 ; 0x08f: 0x00 kfdhdb.siteguid[8]: 0 ; 0x090: 0x00 kfdhdb.siteguid[9]: 0 ; 0x091: 0x00 kfdhdb.siteguid[10]: 0 ; 0x092: 0x00 kfdhdb.siteguid[11]: 0 ; 0x093: 0x00 kfdhdb.siteguid[12]: 0 ; 0x094: 0x00 kfdhdb.siteguid[13]: 0 ; 0x095: 0x00 kfdhdb.siteguid[14]: 0 ; 0x096: 0x00 kfdhdb.siteguid[15]: 0 ; 0x097: 0x00
Now, we create a copy from the dump files generated and substitute the numbers beforely said (the column before “;” and the last two numbers in each line) and put the results into the “mod” directory:
[root@pepito1 kfed]# for i in `ls orig/`; do awk '{ if ($1 ~ /^kfdhdb.siteguid/) printf "%-37s %s %s %s %s\n",$1,"0",$3,$4,"0x00"; else print;}' /home/grid/arumel/kfed/orig/$i > mod/$i; done
You can compare the differences among the old files and the new files created with the information modified:
[root@pepito1 kfed]# for i in `ls orig/`; do diff orig/$i mod/$i ;done
Now, we stop de clusterware in all nodes that belong to the cluster:
[root@pepito1 kfed]# crsctl stop crs [root@pepito2 ~]# crsctl stop crs [root@pepito3 ~]# crsctl stop crs
And we disable the AFD in one of our nodes (it is necessary for doing the kfed ‘write’ operation and let to write on the asm disks):
[root@pepito1 kfed]# asmcmd afd_filter -d
We upload the modified information with the new files into the asm disks:
[root@pepito1 kfed]# bash carga_dmps.sh Uploading the file /home/grid/arumel/kfed/mod/PRODATA101.dump ................ kfedDiscover: Discovering device kfedDiscover: /dev/mapper/PRODATA101p1 is a UFS disk kfedDevOpen: Opening device /dev/mapper/PRODATA101p1 from previous discovery NOTE: kfedOperation : op=write dev=/dev/mapper/PRODATA101p1 text=/home/grid/arumel/kfed/mod/PRODATA101.dump chksum=yes verb=yes (RESULT : Success) Uploading the file /home/grid/arumel/kfed/mod/PRODATA102.dump ................ kfedDiscover: Discovering device kfedDiscover: /dev/mapper/PRODATA102p1 is a UFS disk kfedDevOpen: Opening device /dev/mapper/PRODATA102p1 from previous discovery NOTE: kfedOperation : op=write dev=/dev/mapper/PRODATA102p1 text=/home/grid/arumel/kfed/mod/PRODATA102.dump chksum=yes verb=yes (RESULT : Success) [...]
We enable the AFD in the node:
[root@pepito1 kfed]# asmcmd afd_filter -e
And start the cluster in all the nodes that belong to the cluster:
[root@pepito1 kfed]# crsctl start crs [root@pepito2 ~]# crsctl start crs [root@pepito3 ~]# crsctl start crs
And now, we have removed the information about “sites” in the asm disks without recreate the diskgroups!!!:
SYS@+ASM1> col path format a20
SYS@+ASM1> col failgroup format a20
SYS@+ASM1> col name format a20
SYS@+ASM1> col site_name format a20
SYS@+ASM1> select INST_ID,GROUP_NUMBER,DISK_NUMBER,MOUNT_STATUS,HEADER_STATUS,MODE_STATUS,STATE,NAME,FAILGROUP,SITE_NAME,PATH from gv$asm_disk order by 1,2,3;
INST_ID GROUP_NUMBER DISK_NUMBER MOUNT_S HEADER_STATU MODE_ST STATE NAME FAILGROUP SITE_NAME PATH
---------- ------------ ----------- ------- ------------ ------- -------- -------------------- -------------------- -------------------- --------------------
[...]
1 3 0 CACHED MEMBER ONLINE NORMAL PROMGMTDB101 FGPROMGMTDB1 AFD:PROMGMTDB101
1 3 1 CACHED MEMBER ONLINE NORMAL PROMGMTDB202 FGPROMGMTDB2 AFD:PROMGMTDB202
1 3 2 CACHED MEMBER ONLINE NORMAL PROMGMTDB201 FGPROMGMTDB2 AFD:PROMGMTDB201
1 3 3 CACHED MEMBER ONLINE NORMAL PROMGMTDB102 FGPROMGMTDB1 AFD:PROMGMTDB102
[...]
1 4 0 CACHED MEMBER ONLINE NORMAL PROADMIN101 FGPROADMIN1 AFD:PROADMIN101
1 4 1 CACHED MEMBER ONLINE NORMAL PROADMIN102 FGPROADMIN1 AFD:PROADMIN102
1 4 2 CACHED MEMBER ONLINE NORMAL PROADMIN103 FGPROADMIN1 AFD:PROADMIN103
1 4 3 CACHED MEMBER ONLINE NORMAL PROADMIN104 FGPROADMIN1 AFD:PROADMIN104
1 4 4 CACHED MEMBER ONLINE NORMAL PROADMIN201 FGPROADMIN2 AFD:PROADMIN201
1 4 5 CACHED MEMBER ONLINE NORMAL PROADMIN202 FGPROADMIN2 AFD:PROADMIN202
1 4 6 CACHED MEMBER ONLINE NORMAL PROADMIN203 FGPROADMIN2 AFD:PROADMIN203
1 4 7 CACHED MEMBER ONLINE NORMAL PROADMIN204 FGPROADMIN2 AFD:PROADMIN204
1 5 0 CACHED MEMBER ONLINE NORMAL PROCTLRDO101 FGPROCTLRDO1 AFD:PROCTLRDO101
1 5 1 CACHED MEMBER ONLINE NORMAL PROCTLRDO102 FGPROCTLRDO1 AFD:PROCTLRDO102
1 5 2 CACHED MEMBER ONLINE NORMAL PROCTLRDO103 FGPROCTLRDO1 AFD:PROCTLRDO103
1 5 3 CACHED MEMBER ONLINE NORMAL PROCTLRDO104 FGPROCTLRDO1 AFD:PROCTLRDO104
1 5 4 CACHED MEMBER ONLINE NORMAL PROCTLRDO105 FGPROCTLRDO1 AFD:PROCTLRDO105
1 5 5 CACHED MEMBER ONLINE NORMAL PROCTLRDO106 FGPROCTLRDO1 AFD:PROCTLRDO106
1 5 6 CACHED MEMBER ONLINE NORMAL PROCTLRDO201 FGPROCTLRDO2 AFD:PROCTLRDO201
1 5 7 CACHED MEMBER ONLINE NORMAL PROCTLRDO202 FGPROCTLRDO2 AFD:PROCTLRDO202
1 5 8 CACHED MEMBER ONLINE NORMAL PROCTLRDO203 FGPROCTLRDO2 AFD:PROCTLRDO203
1 5 9 CACHED MEMBER ONLINE NORMAL PROCTLRDO204 FGPROCTLRDO2 AFD:PROCTLRDO204
1 5 10 CACHED MEMBER ONLINE NORMAL PROCTLRDO205 FGPROCTLRDO2 AFD:PROCTLRDO205
1 5 11 CACHED MEMBER ONLINE NORMAL PROCTLRDO206 FGPROCTLRDO2 AFD:PROCTLRDO206
1 6 0 CACHED MEMBER ONLINE NORMAL PRODATA101 FGPRODATA1 AFD:PRODATA101
1 6 1 CACHED MEMBER ONLINE NORMAL PRODATA202 FGPRODATA2 AFD:PRODATA202
1 6 2 CACHED MEMBER ONLINE NORMAL PRODATA103 FGPRODATA1 AFD:PRODATA103
1 6 3 CACHED MEMBER ONLINE NORMAL PRODATA104 FGPRODATA1 AFD:PRODATA104
1 6 4 CACHED MEMBER ONLINE NORMAL PRODATA105 FGPRODATA1 AFD:PRODATA105
1 6 5 CACHED MEMBER ONLINE NORMAL PRODATA106 FGPRODATA1 AFD:PRODATA106
1 6 6 CACHED MEMBER ONLINE NORMAL PRODATA203 FGPRODATA2 AFD:PRODATA203
1 6 7 CACHED MEMBER ONLINE NORMAL PRODATA204 FGPRODATA2 AFD:PRODATA204
1 6 8 CACHED MEMBER ONLINE NORMAL PRODATA201 FGPRODATA2 AFD:PRODATA201
1 6 9 CACHED MEMBER ONLINE NORMAL PRODATA107 FGPRODATA1 AFD:PRODATA107
1 6 10 CACHED MEMBER ONLINE NORMAL PRODATA102 FGPRODATA1 AFD:PRODATA102
1 6 11 CACHED MEMBER ONLINE NORMAL PRODATA108 FGPRODATA1 AFD:PRODATA108
1 6 12 CACHED MEMBER ONLINE NORMAL PRODATA109 FGPRODATA1 AFD:PRODATA109
1 6 13 CACHED MEMBER ONLINE NORMAL PRODATA205 FGPRODATA2 AFD:PRODATA205
1 6 14 CACHED MEMBER ONLINE NORMAL PRODATA206 FGPRODATA2 AFD:PRODATA206
1 6 15 CACHED MEMBER ONLINE NORMAL PRODATA207 FGPRODATA2 AFD:PRODATA207
1 6 16 CACHED MEMBER ONLINE NORMAL PRODATA208 FGPRODATA2 AFD:PRODATA208
1 6 17 CACHED MEMBER ONLINE NORMAL PRODATA209 FGPRODATA2 AFD:PRODATA209
1 7 0 CACHED MEMBER ONLINE NORMAL PROFRA101 FGPROFRA1 AFD:PROFRA101
1 7 1 CACHED MEMBER ONLINE NORMAL PROFRA202 FGPROFRA2 AFD:PROFRA202
1 7 2 CACHED MEMBER ONLINE NORMAL PROFRA103 FGPROFRA1 AFD:PROFRA103
1 7 3 CACHED MEMBER ONLINE NORMAL PROFRA104 FGPROFRA1 AFD:PROFRA104
1 7 4 CACHED MEMBER ONLINE NORMAL PROFRA105 FGPROFRA1 AFD:PROFRA105
1 7 5 CACHED MEMBER ONLINE NORMAL PROFRA106 FGPROFRA1 AFD:PROFRA106
1 7 6 CACHED MEMBER ONLINE NORMAL PROFRA201 FGPROFRA2 AFD:PROFRA201
1 7 7 CACHED MEMBER ONLINE NORMAL PROFRA107 FGPROFRA1 AFD:PROFRA107
1 7 8 CACHED MEMBER ONLINE NORMAL PROFRA203 FGPROFRA2 AFD:PROFRA203
1 7 9 CACHED MEMBER ONLINE NORMAL PROFRA108 FGPROFRA1 AFD:PROFRA108
1 7 10 CACHED MEMBER ONLINE NORMAL PROFRA109 FGPROFRA1 AFD:PROFRA109
1 7 11 CACHED MEMBER ONLINE NORMAL PROFRA110 FGPROFRA1 AFD:PROFRA110
1 7 12 CACHED MEMBER ONLINE NORMAL PROFRA102 FGPROFRA1 AFD:PROFRA102
1 7 13 CACHED MEMBER ONLINE NORMAL PROFRA204 FGPROFRA2 AFD:PROFRA204
1 7 14 CACHED MEMBER ONLINE NORMAL PROFRA205 FGPROFRA2 AFD:PROFRA205
1 7 15 CACHED MEMBER ONLINE NORMAL PROFRA206 FGPROFRA2 AFD:PROFRA206
1 7 16 CACHED MEMBER ONLINE NORMAL PROFRA207 FGPROFRA2 AFD:PROFRA207
1 7 17 CACHED MEMBER ONLINE NORMAL PROFRA208 FGPROFRA2 AFD:PROFRA208
1 7 18 CACHED MEMBER ONLINE NORMAL PROFRA209 FGPROFRA2 AFD:PROFRA209
1 7 19 CACHED MEMBER ONLINE NORMAL PROFRA210 FGPROFRA2 AFD:PROFRA210
1 8 0 CACHED MEMBER ONLINE NORMAL VOTOCR101 FGVOTOCR101 site1 AFD:VOTOCR101
1 8 1 CACHED MEMBER ONLINE NORMAL VOTOCR102 FGVOTOCR102 site1 AFD:VOTOCR102
1 8 2 CACHED MEMBER ONLINE NORMAL VOTOCR103 FGVOTOCR103 site1 AFD:VOTOCR103
1 8 3 CACHED MEMBER ONLINE NORMAL VOTOCR201 FGVOTOCR201 site2 AFD:VOTOCR201
1 8 4 CACHED MEMBER ONLINE NORMAL VOTOCR202 FGVOTOCR202 site2 AFD:VOTOCR202
1 8 5 CACHED MEMBER ONLINE NORMAL VOTOCR203 FGVOTOCR203 site2 AFD:VOTOCR203
1 8 6 CACHED MEMBER ONLINE NORMAL VOTOCR301 FGVOTOCR301 site3 AFD:VOTOCR301
83 rows selected.
Adding value with Arumel!!
