Cómo sustituir el “site” en asm de unos discos en un diskgroup con redundancia normal

 

(Antes de nada, agradecer la colaboración de Óscar, mi GRAN amigo, para crear el script de modificación con awk)

 

En una versión 12.2 de clusterware, tenemos configurado el AFD (asm filter driver) y algunos discos pertenecen, por error, a unos sites, ya que se configuró un cluster extendido pero no se están usando extended diskgroups salvo para el voting y ocr.

Algunos discos que salen en la consulta, son tras unas pruebas que hemos estado haciendo de split-brain y que ya explicaremos en otra entrada de blog el resultado. Por lo que podemos centrarnos únicamente en la columna “site_name”.

SYS@+ASM1> col path format a20
SYS@+ASM1> col failgroup format a20
SYS@+ASM1> col name format a20
SYS@+ASM1> col site_name format a20
SYS@+ASM1> select INST_ID,GROUP_NUMBER,DISK_NUMBER,MOUNT_STATUS,HEADER_STATUS,MODE_STATUS,STATE,NAME,FAILGROUP,SITE_NAME,PATH from gv$asm_disk order by 1,2,3;

   INST_ID GROUP_NUMBER DISK_NUMBER MOUNT_S HEADER_STATU MODE_ST STATE    NAME                 FAILGROUP            SITE_NAME PATH
---------- ------------ ----------- ------- ------------ ------- -------- -------------------- -------------------- --------- --------------------
[...]
         1            3           0 CACHED  MEMBER       ONLINE  NORMAL   PROMGMTDB101         FGPROMGMTDB1         site1     AFD:PROMGMTDB101
         1            3           1 CACHED  MEMBER       ONLINE  NORMAL   PROMGMTDB202         FGPROMGMTDB2         site2     AFD:PROMGMTDB202
         1            3           2 CACHED  MEMBER       ONLINE  NORMAL   PROMGMTDB201         FGPROMGMTDB2         site2     AFD:PROMGMTDB201
         1            3           3 CACHED  MEMBER       ONLINE  NORMAL   PROMGMTDB102         FGPROMGMTDB1         site1     AFD:PROMGMTDB102
[...]
         1            6           0 CACHED  MEMBER       ONLINE  NORMAL   PRODATA101           FGPRODATA1           site1     AFD:PRODATA101
         1            6           1 CACHED  MEMBER       ONLINE  NORMAL   PRODATA202           FGPRODATA2           site2     AFD:PRODATA202
         1            6           2 CACHED  MEMBER       ONLINE  NORMAL   PRODATA103           FGPRODATA1           site1     AFD:PRODATA103
         1            6           3 CACHED  MEMBER       ONLINE  NORMAL   PRODATA104           FGPRODATA1           site1     AFD:PRODATA104
         1            6           4 CACHED  MEMBER       ONLINE  NORMAL   PRODATA105           FGPRODATA1           site1     AFD:PRODATA105
         1            6           5 CACHED  MEMBER       ONLINE  NORMAL   PRODATA106           FGPRODATA1           site1     AFD:PRODATA106
         1            6           6 CACHED  MEMBER       ONLINE  NORMAL   PRODATA203           FGPRODATA2           site2     AFD:PRODATA203
         1            6           7 CACHED  MEMBER       ONLINE  NORMAL   PRODATA204           FGPRODATA2           site2     AFD:PRODATA204
         1            6           8 CACHED  MEMBER       ONLINE  NORMAL   PRODATA201           FGPRODATA2           site2     AFD:PRODATA201
         1            6           9 CACHED  MEMBER       ONLINE  NORMAL   PRODATA107           FGPRODATA1           site1     AFD:PRODATA107
         1            6          10 CACHED  MEMBER       ONLINE  NORMAL   PRODATA102           FGPRODATA1           site1     AFD:PRODATA102
         1            6          11 CACHED  MEMBER       ONLINE  NORMAL   PRODATA108           FGPRODATA1           site1     AFD:PRODATA108
         1            6          12 CACHED  MEMBER       ONLINE  NORMAL   PRODATA109           FGPRODATA1           site1     AFD:PRODATA109
         1            6          13 CACHED  MEMBER       ONLINE  NORMAL   PRODATA205           FGPRODATA2           site2     AFD:PRODATA205
         1            6          14 CACHED  MEMBER       ONLINE  NORMAL   PRODATA206           FGPRODATA2           site2     AFD:PRODATA206
         1            6          15 CACHED  MEMBER       ONLINE  NORMAL   PRODATA207           FGPRODATA2           site2     AFD:PRODATA207
         1            6          16 CACHED  MEMBER       ONLINE  NORMAL   PRODATA208           FGPRODATA2           site2     AFD:PRODATA208
         1            6          17 CACHED  MEMBER       ONLINE  NORMAL   PRODATA209           FGPRODATA2           site2     AFD:PRODATA209
         1            7           0 CACHED  MEMBER       ONLINE  NORMAL   PROFRA101            FGPROFRA1            site1     AFD:PROFRA101
         1            7           1 CACHED  MEMBER       ONLINE  NORMAL   PROFRA202            FGPROFRA2            site2     AFD:PROFRA202
         1            7           2 CACHED  MEMBER       ONLINE  NORMAL   PROFRA103            FGPROFRA1            site1     AFD:PROFRA103
         1            7           3 CACHED  MEMBER       ONLINE  NORMAL   PROFRA104            FGPROFRA1            site1     AFD:PROFRA104
         1            7           4 CACHED  MEMBER       ONLINE  NORMAL   PROFRA105            FGPROFRA1            site1     AFD:PROFRA105
         1            7           5 CACHED  MEMBER       ONLINE  NORMAL   PROFRA106            FGPROFRA1            site1     AFD:PROFRA106
         1            7           6 CACHED  MEMBER       ONLINE  NORMAL   PROFRA201            FGPROFRA2            site2     AFD:PROFRA201
         1            7           7 CACHED  MEMBER       ONLINE  NORMAL   PROFRA107            FGPROFRA1            site1     AFD:PROFRA107
         1            7           8 CACHED  MEMBER       ONLINE  NORMAL   PROFRA203            FGPROFRA2            site2     AFD:PROFRA203
         1            7           9 CACHED  MEMBER       ONLINE  NORMAL   PROFRA108            FGPROFRA1            site1     AFD:PROFRA108
         1            7          10 CACHED  MEMBER       ONLINE  NORMAL   PROFRA109            FGPROFRA1            site1     AFD:PROFRA109
         1            7          11 CACHED  MEMBER       ONLINE  NORMAL   PROFRA110            FGPROFRA1            site1     AFD:PROFRA110
         1            7          12 CACHED  MEMBER       ONLINE  NORMAL   PROFRA102            FGPROFRA1            site1     AFD:PROFRA102
         1            7          13 CACHED  MEMBER       ONLINE  NORMAL   PROFRA204            FGPROFRA2            site2     AFD:PROFRA204
         1            7          14 CACHED  MEMBER       ONLINE  NORMAL   PROFRA205            FGPROFRA2            site2     AFD:PROFRA205
         1            7          15 CACHED  MEMBER       ONLINE  NORMAL   PROFRA206            FGPROFRA2            site2     AFD:PROFRA206
         1            7          16 CACHED  MEMBER       ONLINE  NORMAL   PROFRA207            FGPROFRA2            site2     AFD:PROFRA207
         1            7          17 CACHED  MEMBER       ONLINE  NORMAL   PROFRA208            FGPROFRA2            site2     AFD:PROFRA208
         1            7          18 CACHED  MEMBER       ONLINE  NORMAL   PROFRA209            FGPROFRA2            site2     AFD:PROFRA209
         1            7          19 CACHED  MEMBER       ONLINE  NORMAL   PROFRA210            FGPROFRA2            site2     AFD:PROFRA210
         1            8           0 CACHED  MEMBER       ONLINE  NORMAL   VOTOCR101            FGVOTOCR101          site1     AFD:VOTOCR101
         1            8           1 CACHED  MEMBER       ONLINE  NORMAL   VOTOCR102            FGVOTOCR102          site1     AFD:VOTOCR102
         1            8           2 CACHED  MEMBER       ONLINE  NORMAL   VOTOCR103            FGVOTOCR103          site1     AFD:VOTOCR103
         1            8           3 CACHED  MEMBER       ONLINE  NORMAL   VOTOCR201            FGVOTOCR201          site2     AFD:VOTOCR201
         1            8           4 CACHED  MEMBER       ONLINE  NORMAL   VOTOCR202            FGVOTOCR202          site2     AFD:VOTOCR202
         1            8           5 CACHED  MEMBER       ONLINE  NORMAL   VOTOCR203            FGVOTOCR203          site2     AFD:VOTOCR203
         1            8           6 CACHED  MEMBER       ONLINE  NORMAL   VOTOCR301            FGVOTOCR301          site3     AFD:VOTOCR301

83 rows selected.

 

Vamos a suprimirlo, no a cambiarlo o añadirlo, por lo que no usamos la opción que nos ofrece el kfed:

si/telbl KFED Site Label to stamp [sitelbl=string]

 

Nos creamos un directorio donde descargaremos los dumps del metadato de cada disco, y otro directorio donde los modificaremos, por si tenemos que dar marcha atrás tener el dato original: 😉

[root@pepito1 ~]# mkdir -p /home/grid/arumel/kfed/orig
[root@pepito1 ~]# mkdir -p /home/grid/arumel/kfed/mod

 

Creamos el script que generará los dumps, quitando el diskgroup que contiene los ocr y voting (en este caso el 8) que sí que tiene que tener el site configurado:

spool /home/grid/arumel/kfed/genera_dmps.sh
select 'echo "Generando el fichero /home/grid/arumel/kfed/orig/' || name || '.dump ................"
kfed op=read dev=/dev/mapper/' || name || 'p1 text=/home/grid/arumel/kfed/orig/' || name || '.dump chksum=yes verb=yes' 
from v$asm_disk 
where site_name is not null 
and group_number!=8 
order by name;
spool off

 

La salida de este script nos creará, unas líneas como las siguientes:

[...]
echo "Generando el fichero /home/grid/arumel/kfed/orig/PRODATA101.dump ................" 
kfed op=read dev=/dev/mapper/PRODATA101p1 text=/home/grid/arumel/kfed/orig/PRODATA101.dump chksum=yes verb=yes 
[...]

 

Creamos el script que cargará los datos modificados en los discos otra vez (importante notar que el directorio ahora es el “mod”!!)

spool /home/grid/arumel/kfed/carga_dmps.sh
select 'echo "Cargando el fichero /home/grid/arumel/kfed/mod/' || name || '.dump ................"
kfed op=write dev=/dev/mapper/' || name || 'p1 text=/home/grid/arumel/kfed/mod/' || name || '.dump chksum=yes verb=yes' 
from v$asm_disk 
where site_name is not null 
and group_number!=8 
order by name;
spool off

 

La salida de este script, nos creará unas líneas como las siguientes:

[...]
echo "Cargando el fichero /home/grid/arumel/kfed/mod/PRODATA101.dump ................" 
kfed op=write dev=/dev/mapper/PRODATA101p1 text=/home/grid/arumel/kfed/mod/PRODATA101.dump chksum=yes verb=yes 
[...]

 

Comprobamos los ficheros creados:

[root@pepito1 ~]# cd /home/grid/arumel/kfed/
[root@pepito1 kfed]# ls
carga_dmps.sh genera_dmps.sh mod orig

 

Lanzamos como root el script que genera los ficheros dumps de nuestros discos asm (para la operación ‘read’ del kfed no le interfiere que el afd esté activado):

[root@ctbl109 kfed]# bash genera_dmps.sh
Generando el fichero /home/grid/arumel/kfed/orig/PRODATA101.dump ................
kfedDiscover: Discovering device
kfedDiscover: /dev/mapper/PRODATA101p1 is a UFS disk
kfedDevOpen: Opening device /dev/mapper/PRODATA101p1 from previous discovery
NOTE: kfedOperation : op=read dev=/dev/mapper/PRODATA101p1 text=/home/grid/arumel/kfed/orig/PRODATA101.dump chksum=yes verb=yes (RESULT : Success)
Generando el fichero /home/grid/arumel/kfed/orig/PRODATA102.dump ................
kfedDiscover: Discovering device
kfedDiscover: /dev/mapper/PRODATA102p1 is a UFS disk
kfedDevOpen: Opening device /dev/mapper/PRODATA102p1 from previous discovery
NOTE: kfedOperation : op=read dev=/dev/mapper/PRODATA102p1 text=/home/grid/arumel/kfed/orig/PRODATA102.dump chksum=yes verb=yes (RESULT : Success)
[...]

 

Comparando la salida de un disco sin “site” configurado con uno que sí que lo tiene, vemos que la diferencia está en los dos últimos dígitos de cada línea, que coincidirán con el siteGUID del comando “crsctl query cluster site -all” por ejemplo

Site 'site1' identified by GUID '37c347fd2c83ffb9bf127d0aca787ca9' in state 'ENABLED' [...]

 

también nos fijamos en que no aparece a “0” la columna antes del “;” como pasará con uno que no lo tenga configurado:

[root@pepito1 kfed]# grep siteguid prueba.dmp 
kfdhdb.siteguid[0]:                  55 ; 0x088: 0x37
kfdhdb.siteguid[1]:                 195 ; 0x089: 0xc3
kfdhdb.siteguid[2]:                  71 ; 0x08a: 0x47
kfdhdb.siteguid[3]:                 253 ; 0x08b: 0xfd
kfdhdb.siteguid[4]:                  44 ; 0x08c: 0x2c
kfdhdb.siteguid[5]:                 131 ; 0x08d: 0x83
kfdhdb.siteguid[6]:                 255 ; 0x08e: 0xff
kfdhdb.siteguid[7]:                 185 ; 0x08f: 0xb9
kfdhdb.siteguid[8]:                 191 ; 0x090: 0xbf
kfdhdb.siteguid[9]:                  18 ; 0x091: 0x12
kfdhdb.siteguid[10]:                125 ; 0x092: 0x7d
kfdhdb.siteguid[11]:                 10 ; 0x093: 0x0a
kfdhdb.siteguid[12]:                202 ; 0x094: 0xca
kfdhdb.siteguid[13]:                120 ; 0x095: 0x78
kfdhdb.siteguid[14]:                124 ; 0x096: 0x7c
kfdhdb.siteguid[15]:                169 ; 0x097: 0xa9

 

En uno que no lo tiene configurado, debería estar:

[root@pepito1 kfed]# grep siteguid prueba2.dmp 
kfdhdb.siteguid[0]:                   0 ; 0x088: 0x00
kfdhdb.siteguid[1]:                   0 ; 0x089: 0x00
kfdhdb.siteguid[2]:                   0 ; 0x08a: 0x00
kfdhdb.siteguid[3]:                   0 ; 0x08b: 0x00
kfdhdb.siteguid[4]:                   0 ; 0x08c: 0x00
kfdhdb.siteguid[5]:                   0 ; 0x08d: 0x00
kfdhdb.siteguid[6]:                   0 ; 0x08e: 0x00
kfdhdb.siteguid[7]:                   0 ; 0x08f: 0x00
kfdhdb.siteguid[8]:                   0 ; 0x090: 0x00
kfdhdb.siteguid[9]:                   0 ; 0x091: 0x00
kfdhdb.siteguid[10]:                  0 ; 0x092: 0x00
kfdhdb.siteguid[11]:                  0 ; 0x093: 0x00
kfdhdb.siteguid[12]:                  0 ; 0x094: 0x00
kfdhdb.siteguid[13]:                  0 ; 0x095: 0x00
kfdhdb.siteguid[14]:                  0 ; 0x096: 0x00
kfdhdb.siteguid[15]:                  0 ; 0x097: 0x00

 

 

Nos creamos una copia de los dumps generados sustituyendo las entradas donde aparece el “siteguid” y dejamos estos ficheros nuevos en el directorio “mod”, creado anteriormente:

[root@pepito1 kfed]# for i in `ls orig/`; do awk '{ if ($1 ~ /^kfdhdb.siteguid/) printf "%-37s %s %s %s %s\n",$1,"0",$3,$4,"0x00"; else print;}' /home/grid/arumel/kfed/orig/$i > mod/$i; done

 

Comparamos las diferencias entre los ficheros antiguos y nuevos, con la información original y la nueva modificada:

[root@pepito1 kfed]# for i in `ls orig/`; do diff orig/$i mod/$i ;done

 

Paramos el clusterware en todos los nodos del cluster:

[root@pepito1 kfed]# crsctl stop crs
[root@pepito2 ~]# crsctl stop crs
[root@pepito3 ~]# crsctl stop crs

 

Deshabilitamos el AFD en uno de los nodos (es necesario para que la operación ‘write’ del kfed permita escribir sobre los discos de asm):

[root@pepito1 kfed]# asmcmd afd_filter -d

 

Cargamos los nuevos ficheros modificados:

[root@pepito1 kfed]# bash carga_dmps.sh 
Cargando el fichero /home/grid/arumel/kfed/mod/PRODATA101.dump ................
kfedDiscover: Discovering device
kfedDiscover: /dev/mapper/PRODATA101p1 is a UFS disk
kfedDevOpen: Opening device /dev/mapper/PRODATA101p1 from previous discovery
NOTE: kfedOperation : op=write dev=/dev/mapper/PRODATA101p1 text=/home/grid/arumel/kfed/mod/PRODATA101.dump chksum=yes verb=yes (RESULT : Success)
Cargando el fichero /home/grid/arumel/kfed/mod/PRODATA102.dump ................
kfedDiscover: Discovering device
kfedDiscover: /dev/mapper/PRODATA102p1 is a UFS disk
kfedDevOpen: Opening device /dev/mapper/PRODATA102p1 from previous discovery
NOTE: kfedOperation : op=write dev=/dev/mapper/PRODATA102p1 text=/home/grid/arumel/kfed/mod/PRODATA102.dump chksum=yes verb=yes (RESULT : Success)
[...]

 

Habilitamos el AFD en el nodo que lo habíamos deshabilitado:

[root@pepito1 kfed]# asmcmd afd_filter -e

 

Arrancamos el clusterware en todos los nodos del cluster:

[root@pepito1 kfed]# crsctl start crs
[root@pepito2 ~]# crsctl start crs 
[root@pepito3 ~]# crsctl start crs

 

Y ya tenemos eliminados los discos con los sites!!!:

SYS@+ASM1> col path format a20
SYS@+ASM1> col failgroup format a20
SYS@+ASM1> col name format a20
SYS@+ASM1> col site_name format a20
SYS@+ASM1> select INST_ID,GROUP_NUMBER,DISK_NUMBER,MOUNT_STATUS,HEADER_STATUS,MODE_STATUS,STATE,NAME,FAILGROUP,SITE_NAME,PATH from gv$asm_disk order by 1,2,3;

   INST_ID GROUP_NUMBER DISK_NUMBER MOUNT_S HEADER_STATU MODE_ST STATE    NAME                 FAILGROUP            SITE_NAME            PATH
---------- ------------ ----------- ------- ------------ ------- -------- -------------------- -------------------- -------------------- --------------------
[...]
         1            1           0 CACHED  MEMBER       ONLINE  NORMAL   DGACFS101            FGSITE1                                   AFD:TESTSPLIT101
         1            1           1 CACHED  MEMBER       ONLINE  NORMAL   DGACFS102            FGSITE1                                   AFD:TESTSPLIT102
         1            1           2 CACHED  MEMBER       ONLINE  NORMAL   DGACFS201            FGSITE2                                   AFD:TESTSPLIT201
         1            1           3 CACHED  MEMBER       ONLINE  NORMAL   DGACFS202            FGSITE2                                   AFD:TESTSPLIT202
         1            2           0 CACHED  MEMBER       ONLINE  NORMAL   DGASM101             FGSITE1                                   AFD:TESTEXT101
         1            2           1 CACHED  MEMBER       ONLINE  NORMAL   DGASM102             FGSITE1                                   AFD:TESTEXT102
         1            2           4 CACHED  MEMBER       ONLINE  NORMAL   TESTEXT201           FGSITE2                                   AFD:TESTEXT201
         1            2           5 CACHED  MEMBER       ONLINE  NORMAL   TESTEXT202           FGSITE2                                   AFD:TESTEXT202
         1            3           0 CACHED  MEMBER       ONLINE  NORMAL   PROMGMTDB101         FGPROMGMTDB1                              AFD:PROMGMTDB101
         1            3           1 CACHED  MEMBER       ONLINE  NORMAL   PROMGMTDB202         FGPROMGMTDB2                              AFD:PROMGMTDB202
         1            3           2 CACHED  MEMBER       ONLINE  NORMAL   PROMGMTDB201         FGPROMGMTDB2                              AFD:PROMGMTDB201
         1            3           3 CACHED  MEMBER       ONLINE  NORMAL   PROMGMTDB102         FGPROMGMTDB1                              AFD:PROMGMTDB102
         1            4           0 CACHED  MEMBER       ONLINE  NORMAL   PROADMIN101          FGPROADMIN1                               AFD:PROADMIN101
         1            4           1 CACHED  MEMBER       ONLINE  NORMAL   PROADMIN102          FGPROADMIN1                               AFD:PROADMIN102
         1            4           2 CACHED  MEMBER       ONLINE  NORMAL   PROADMIN103          FGPROADMIN1                               AFD:PROADMIN103
         1            4           3 CACHED  MEMBER       ONLINE  NORMAL   PROADMIN104          FGPROADMIN1                               AFD:PROADMIN104
         1            4           4 CACHED  MEMBER       ONLINE  NORMAL   PROADMIN201          FGPROADMIN2                               AFD:PROADMIN201
         1            4           5 CACHED  MEMBER       ONLINE  NORMAL   PROADMIN202          FGPROADMIN2                               AFD:PROADMIN202
         1            4           6 CACHED  MEMBER       ONLINE  NORMAL   PROADMIN203          FGPROADMIN2                               AFD:PROADMIN203
         1            4           7 CACHED  MEMBER       ONLINE  NORMAL   PROADMIN204          FGPROADMIN2                               AFD:PROADMIN204
         1            5           0 CACHED  MEMBER       ONLINE  NORMAL   PROCTLRDO101         FGPROCTLRDO1                              AFD:PROCTLRDO101
         1            5           1 CACHED  MEMBER       ONLINE  NORMAL   PROCTLRDO102         FGPROCTLRDO1                              AFD:PROCTLRDO102
         1            5           2 CACHED  MEMBER       ONLINE  NORMAL   PROCTLRDO103         FGPROCTLRDO1                              AFD:PROCTLRDO103
         1            5           3 CACHED  MEMBER       ONLINE  NORMAL   PROCTLRDO104         FGPROCTLRDO1                              AFD:PROCTLRDO104
         1            5           4 CACHED  MEMBER       ONLINE  NORMAL   PROCTLRDO105         FGPROCTLRDO1                              AFD:PROCTLRDO105
         1            5           5 CACHED  MEMBER       ONLINE  NORMAL   PROCTLRDO106         FGPROCTLRDO1                              AFD:PROCTLRDO106
         1            5           6 CACHED  MEMBER       ONLINE  NORMAL   PROCTLRDO201         FGPROCTLRDO2                              AFD:PROCTLRDO201
         1            5           7 CACHED  MEMBER       ONLINE  NORMAL   PROCTLRDO202         FGPROCTLRDO2                              AFD:PROCTLRDO202
         1            5           8 CACHED  MEMBER       ONLINE  NORMAL   PROCTLRDO203         FGPROCTLRDO2                              AFD:PROCTLRDO203
         1            5           9 CACHED  MEMBER       ONLINE  NORMAL   PROCTLRDO204         FGPROCTLRDO2                              AFD:PROCTLRDO204
         1            5          10 CACHED  MEMBER       ONLINE  NORMAL   PROCTLRDO205         FGPROCTLRDO2                              AFD:PROCTLRDO205
         1            5          11 CACHED  MEMBER       ONLINE  NORMAL   PROCTLRDO206         FGPROCTLRDO2                              AFD:PROCTLRDO206
         1            6           0 CACHED  MEMBER       ONLINE  NORMAL   PRODATA101           FGPRODATA1                                AFD:PRODATA101
         1            6           1 CACHED  MEMBER       ONLINE  NORMAL   PRODATA202           FGPRODATA2                                AFD:PRODATA202
         1            6           2 CACHED  MEMBER       ONLINE  NORMAL   PRODATA103           FGPRODATA1                                AFD:PRODATA103
         1            6           3 CACHED  MEMBER       ONLINE  NORMAL   PRODATA104           FGPRODATA1                                AFD:PRODATA104
         1            6           4 CACHED  MEMBER       ONLINE  NORMAL   PRODATA105           FGPRODATA1                                AFD:PRODATA105
         1            6           5 CACHED  MEMBER       ONLINE  NORMAL   PRODATA106           FGPRODATA1                                AFD:PRODATA106
         1            6           6 CACHED  MEMBER       ONLINE  NORMAL   PRODATA203           FGPRODATA2                                AFD:PRODATA203
         1            6           7 CACHED  MEMBER       ONLINE  NORMAL   PRODATA204           FGPRODATA2                                AFD:PRODATA204
         1            6           8 CACHED  MEMBER       ONLINE  NORMAL   PRODATA201           FGPRODATA2                                AFD:PRODATA201
         1            6           9 CACHED  MEMBER       ONLINE  NORMAL   PRODATA107           FGPRODATA1                                AFD:PRODATA107
         1            6          10 CACHED  MEMBER       ONLINE  NORMAL   PRODATA102           FGPRODATA1                                AFD:PRODATA102
         1            6          11 CACHED  MEMBER       ONLINE  NORMAL   PRODATA108           FGPRODATA1                                AFD:PRODATA108
         1            6          12 CACHED  MEMBER       ONLINE  NORMAL   PRODATA109           FGPRODATA1                                AFD:PRODATA109
         1            6          13 CACHED  MEMBER       ONLINE  NORMAL   PRODATA205           FGPRODATA2                                AFD:PRODATA205
         1            6          14 CACHED  MEMBER       ONLINE  NORMAL   PRODATA206           FGPRODATA2                                AFD:PRODATA206
         1            6          15 CACHED  MEMBER       ONLINE  NORMAL   PRODATA207           FGPRODATA2                                AFD:PRODATA207
         1            6          16 CACHED  MEMBER       ONLINE  NORMAL   PRODATA208           FGPRODATA2                                AFD:PRODATA208
         1            6          17 CACHED  MEMBER       ONLINE  NORMAL   PRODATA209           FGPRODATA2                                AFD:PRODATA209
         1            7           0 CACHED  MEMBER       ONLINE  NORMAL   PROFRA101            FGPROFRA1                                 AFD:PROFRA101
         1            7           1 CACHED  MEMBER       ONLINE  NORMAL   PROFRA202            FGPROFRA2                                 AFD:PROFRA202
         1            7           2 CACHED  MEMBER       ONLINE  NORMAL   PROFRA103            FGPROFRA1                                 AFD:PROFRA103
         1            7           3 CACHED  MEMBER       ONLINE  NORMAL   PROFRA104            FGPROFRA1                                 AFD:PROFRA104
         1            7           4 CACHED  MEMBER       ONLINE  NORMAL   PROFRA105            FGPROFRA1                                 AFD:PROFRA105
         1            7           5 CACHED  MEMBER       ONLINE  NORMAL   PROFRA106            FGPROFRA1                                 AFD:PROFRA106
         1            7           6 CACHED  MEMBER       ONLINE  NORMAL   PROFRA201            FGPROFRA2                                 AFD:PROFRA201
         1            7           7 CACHED  MEMBER       ONLINE  NORMAL   PROFRA107            FGPROFRA1                                 AFD:PROFRA107
         1            7           8 CACHED  MEMBER       ONLINE  NORMAL   PROFRA203            FGPROFRA2                                 AFD:PROFRA203
         1            7           9 CACHED  MEMBER       ONLINE  NORMAL   PROFRA108            FGPROFRA1                                 AFD:PROFRA108
         1            7          10 CACHED  MEMBER       ONLINE  NORMAL   PROFRA109            FGPROFRA1                                 AFD:PROFRA109
         1            7          11 CACHED  MEMBER       ONLINE  NORMAL   PROFRA110            FGPROFRA1                                 AFD:PROFRA110
         1            7          12 CACHED  MEMBER       ONLINE  NORMAL   PROFRA102            FGPROFRA1                                 AFD:PROFRA102
         1            7          13 CACHED  MEMBER       ONLINE  NORMAL   PROFRA204            FGPROFRA2                                 AFD:PROFRA204
         1            7          14 CACHED  MEMBER       ONLINE  NORMAL   PROFRA205            FGPROFRA2                                 AFD:PROFRA205
         1            7          15 CACHED  MEMBER       ONLINE  NORMAL   PROFRA206            FGPROFRA2                                 AFD:PROFRA206
         1            7          16 CACHED  MEMBER       ONLINE  NORMAL   PROFRA207            FGPROFRA2                                 AFD:PROFRA207
         1            7          17 CACHED  MEMBER       ONLINE  NORMAL   PROFRA208            FGPROFRA2                                 AFD:PROFRA208
         1            7          18 CACHED  MEMBER       ONLINE  NORMAL   PROFRA209            FGPROFRA2                                 AFD:PROFRA209
         1            7          19 CACHED  MEMBER       ONLINE  NORMAL   PROFRA210            FGPROFRA2                                 AFD:PROFRA210
         1            8           0 CACHED  MEMBER       ONLINE  NORMAL   VOTOCR101            FGVOTOCR101          site1                AFD:VOTOCR101
         1            8           1 CACHED  MEMBER       ONLINE  NORMAL   VOTOCR102            FGVOTOCR102          site1                AFD:VOTOCR102
         1            8           2 CACHED  MEMBER       ONLINE  NORMAL   VOTOCR103            FGVOTOCR103          site1                AFD:VOTOCR103
         1            8           3 CACHED  MEMBER       ONLINE  NORMAL   VOTOCR201            FGVOTOCR201          site2                AFD:VOTOCR201
         1            8           4 CACHED  MEMBER       ONLINE  NORMAL   VOTOCR202            FGVOTOCR202          site2                AFD:VOTOCR202
         1            8           5 CACHED  MEMBER       ONLINE  NORMAL   VOTOCR203            FGVOTOCR203          site2                AFD:VOTOCR203
         1            8           6 CACHED  MEMBER       ONLINE  NORMAL   VOTOCR301            FGVOTOCR301          site3                AFD:VOTOCR301

83 rows selected.

 

 

Añadiendo valor con Arumel!!