Cómo sustituir el “site” en asm de unos discos en un diskgroup con redundancia normal
(Antes de nada, agradecer la colaboración de Óscar, mi GRAN amigo, para crear el script de modificación con awk)
En una versión 12.2 de clusterware, tenemos configurado el AFD (asm filter driver) y algunos discos pertenecen, por error, a unos sites, ya que se configuró un cluster extendido pero no se están usando extended diskgroups salvo para el voting y ocr.
Algunos discos que salen en la consulta, son tras unas pruebas que hemos estado haciendo de split-brain y que ya explicaremos en otra entrada de blog el resultado. Por lo que podemos centrarnos únicamente en la columna “site_name”.
SYS@+ASM1> col path format a20 SYS@+ASM1> col failgroup format a20 SYS@+ASM1> col name format a20 SYS@+ASM1> col site_name format a20 SYS@+ASM1> select INST_ID,GROUP_NUMBER,DISK_NUMBER,MOUNT_STATUS,HEADER_STATUS,MODE_STATUS,STATE,NAME,FAILGROUP,SITE_NAME,PATH from gv$asm_disk order by 1,2,3; INST_ID GROUP_NUMBER DISK_NUMBER MOUNT_S HEADER_STATU MODE_ST STATE NAME FAILGROUP SITE_NAME PATH ---------- ------------ ----------- ------- ------------ ------- -------- -------------------- -------------------- --------- -------------------- [...] 1 3 0 CACHED MEMBER ONLINE NORMAL PROMGMTDB101 FGPROMGMTDB1 site1 AFD:PROMGMTDB101 1 3 1 CACHED MEMBER ONLINE NORMAL PROMGMTDB202 FGPROMGMTDB2 site2 AFD:PROMGMTDB202 1 3 2 CACHED MEMBER ONLINE NORMAL PROMGMTDB201 FGPROMGMTDB2 site2 AFD:PROMGMTDB201 1 3 3 CACHED MEMBER ONLINE NORMAL PROMGMTDB102 FGPROMGMTDB1 site1 AFD:PROMGMTDB102 [...] 1 6 0 CACHED MEMBER ONLINE NORMAL PRODATA101 FGPRODATA1 site1 AFD:PRODATA101 1 6 1 CACHED MEMBER ONLINE NORMAL PRODATA202 FGPRODATA2 site2 AFD:PRODATA202 1 6 2 CACHED MEMBER ONLINE NORMAL PRODATA103 FGPRODATA1 site1 AFD:PRODATA103 1 6 3 CACHED MEMBER ONLINE NORMAL PRODATA104 FGPRODATA1 site1 AFD:PRODATA104 1 6 4 CACHED MEMBER ONLINE NORMAL PRODATA105 FGPRODATA1 site1 AFD:PRODATA105 1 6 5 CACHED MEMBER ONLINE NORMAL PRODATA106 FGPRODATA1 site1 AFD:PRODATA106 1 6 6 CACHED MEMBER ONLINE NORMAL PRODATA203 FGPRODATA2 site2 AFD:PRODATA203 1 6 7 CACHED MEMBER ONLINE NORMAL PRODATA204 FGPRODATA2 site2 AFD:PRODATA204 1 6 8 CACHED MEMBER ONLINE NORMAL PRODATA201 FGPRODATA2 site2 AFD:PRODATA201 1 6 9 CACHED MEMBER ONLINE NORMAL PRODATA107 FGPRODATA1 site1 AFD:PRODATA107 1 6 10 CACHED MEMBER ONLINE NORMAL PRODATA102 FGPRODATA1 site1 AFD:PRODATA102 1 6 11 CACHED MEMBER ONLINE NORMAL PRODATA108 FGPRODATA1 site1 AFD:PRODATA108 1 6 12 CACHED MEMBER ONLINE NORMAL PRODATA109 FGPRODATA1 site1 AFD:PRODATA109 1 6 13 CACHED MEMBER ONLINE NORMAL PRODATA205 FGPRODATA2 site2 AFD:PRODATA205 1 6 14 CACHED MEMBER ONLINE NORMAL PRODATA206 FGPRODATA2 site2 AFD:PRODATA206 1 6 15 CACHED MEMBER ONLINE NORMAL PRODATA207 FGPRODATA2 site2 AFD:PRODATA207 1 6 16 CACHED MEMBER ONLINE NORMAL PRODATA208 FGPRODATA2 site2 AFD:PRODATA208 1 6 17 CACHED MEMBER ONLINE NORMAL PRODATA209 FGPRODATA2 site2 AFD:PRODATA209 1 7 0 CACHED MEMBER ONLINE NORMAL PROFRA101 FGPROFRA1 site1 AFD:PROFRA101 1 7 1 CACHED MEMBER ONLINE NORMAL PROFRA202 FGPROFRA2 site2 AFD:PROFRA202 1 7 2 CACHED MEMBER ONLINE NORMAL PROFRA103 FGPROFRA1 site1 AFD:PROFRA103 1 7 3 CACHED MEMBER ONLINE NORMAL PROFRA104 FGPROFRA1 site1 AFD:PROFRA104 1 7 4 CACHED MEMBER ONLINE NORMAL PROFRA105 FGPROFRA1 site1 AFD:PROFRA105 1 7 5 CACHED MEMBER ONLINE NORMAL PROFRA106 FGPROFRA1 site1 AFD:PROFRA106 1 7 6 CACHED MEMBER ONLINE NORMAL PROFRA201 FGPROFRA2 site2 AFD:PROFRA201 1 7 7 CACHED MEMBER ONLINE NORMAL PROFRA107 FGPROFRA1 site1 AFD:PROFRA107 1 7 8 CACHED MEMBER ONLINE NORMAL PROFRA203 FGPROFRA2 site2 AFD:PROFRA203 1 7 9 CACHED MEMBER ONLINE NORMAL PROFRA108 FGPROFRA1 site1 AFD:PROFRA108 1 7 10 CACHED MEMBER ONLINE NORMAL PROFRA109 FGPROFRA1 site1 AFD:PROFRA109 1 7 11 CACHED MEMBER ONLINE NORMAL PROFRA110 FGPROFRA1 site1 AFD:PROFRA110 1 7 12 CACHED MEMBER ONLINE NORMAL PROFRA102 FGPROFRA1 site1 AFD:PROFRA102 1 7 13 CACHED MEMBER ONLINE NORMAL PROFRA204 FGPROFRA2 site2 AFD:PROFRA204 1 7 14 CACHED MEMBER ONLINE NORMAL PROFRA205 FGPROFRA2 site2 AFD:PROFRA205 1 7 15 CACHED MEMBER ONLINE NORMAL PROFRA206 FGPROFRA2 site2 AFD:PROFRA206 1 7 16 CACHED MEMBER ONLINE NORMAL PROFRA207 FGPROFRA2 site2 AFD:PROFRA207 1 7 17 CACHED MEMBER ONLINE NORMAL PROFRA208 FGPROFRA2 site2 AFD:PROFRA208 1 7 18 CACHED MEMBER ONLINE NORMAL PROFRA209 FGPROFRA2 site2 AFD:PROFRA209 1 7 19 CACHED MEMBER ONLINE NORMAL PROFRA210 FGPROFRA2 site2 AFD:PROFRA210 1 8 0 CACHED MEMBER ONLINE NORMAL VOTOCR101 FGVOTOCR101 site1 AFD:VOTOCR101 1 8 1 CACHED MEMBER ONLINE NORMAL VOTOCR102 FGVOTOCR102 site1 AFD:VOTOCR102 1 8 2 CACHED MEMBER ONLINE NORMAL VOTOCR103 FGVOTOCR103 site1 AFD:VOTOCR103 1 8 3 CACHED MEMBER ONLINE NORMAL VOTOCR201 FGVOTOCR201 site2 AFD:VOTOCR201 1 8 4 CACHED MEMBER ONLINE NORMAL VOTOCR202 FGVOTOCR202 site2 AFD:VOTOCR202 1 8 5 CACHED MEMBER ONLINE NORMAL VOTOCR203 FGVOTOCR203 site2 AFD:VOTOCR203 1 8 6 CACHED MEMBER ONLINE NORMAL VOTOCR301 FGVOTOCR301 site3 AFD:VOTOCR301 83 rows selected.
Vamos a suprimirlo, no a cambiarlo o añadirlo, por lo que no usamos la opción que nos ofrece el kfed:
si/telbl KFED Site Label to stamp [sitelbl=string]
Nos creamos un directorio donde descargaremos los dumps del metadato de cada disco, y otro directorio donde los modificaremos, por si tenemos que dar marcha atrás tener el dato original: 😉
[root@pepito1 ~]# mkdir -p /home/grid/arumel/kfed/orig [root@pepito1 ~]# mkdir -p /home/grid/arumel/kfed/mod
Creamos el script que generará los dumps, quitando el diskgroup que contiene los ocr y voting (en este caso el 8) que sí que tiene que tener el site configurado:
spool /home/grid/arumel/kfed/genera_dmps.sh select 'echo "Generando el fichero /home/grid/arumel/kfed/orig/' || name || '.dump ................" kfed op=read dev=/dev/mapper/' || name || 'p1 text=/home/grid/arumel/kfed/orig/' || name || '.dump chksum=yes verb=yes' from v$asm_disk where site_name is not null and group_number!=8 order by name; spool off
La salida de este script nos creará, unas líneas como las siguientes:
[...] echo "Generando el fichero /home/grid/arumel/kfed/orig/PRODATA101.dump ................" kfed op=read dev=/dev/mapper/PRODATA101p1 text=/home/grid/arumel/kfed/orig/PRODATA101.dump chksum=yes verb=yes [...]
Creamos el script que cargará los datos modificados en los discos otra vez (importante notar que el directorio ahora es el “mod”!!)
spool /home/grid/arumel/kfed/carga_dmps.sh select 'echo "Cargando el fichero /home/grid/arumel/kfed/mod/' || name || '.dump ................" kfed op=write dev=/dev/mapper/' || name || 'p1 text=/home/grid/arumel/kfed/mod/' || name || '.dump chksum=yes verb=yes' from v$asm_disk where site_name is not null and group_number!=8 order by name; spool off
La salida de este script, nos creará unas líneas como las siguientes:
[...] echo "Cargando el fichero /home/grid/arumel/kfed/mod/PRODATA101.dump ................" kfed op=write dev=/dev/mapper/PRODATA101p1 text=/home/grid/arumel/kfed/mod/PRODATA101.dump chksum=yes verb=yes [...]
Comprobamos los ficheros creados:
[root@pepito1 ~]# cd /home/grid/arumel/kfed/ [root@pepito1 kfed]# ls carga_dmps.sh genera_dmps.sh mod orig
Lanzamos como root el script que genera los ficheros dumps de nuestros discos asm (para la operación ‘read’ del kfed no le interfiere que el afd esté activado):
[root@ctbl109 kfed]# bash genera_dmps.sh Generando el fichero /home/grid/arumel/kfed/orig/PRODATA101.dump ................ kfedDiscover: Discovering device kfedDiscover: /dev/mapper/PRODATA101p1 is a UFS disk kfedDevOpen: Opening device /dev/mapper/PRODATA101p1 from previous discovery NOTE: kfedOperation : op=read dev=/dev/mapper/PRODATA101p1 text=/home/grid/arumel/kfed/orig/PRODATA101.dump chksum=yes verb=yes (RESULT : Success) Generando el fichero /home/grid/arumel/kfed/orig/PRODATA102.dump ................ kfedDiscover: Discovering device kfedDiscover: /dev/mapper/PRODATA102p1 is a UFS disk kfedDevOpen: Opening device /dev/mapper/PRODATA102p1 from previous discovery NOTE: kfedOperation : op=read dev=/dev/mapper/PRODATA102p1 text=/home/grid/arumel/kfed/orig/PRODATA102.dump chksum=yes verb=yes (RESULT : Success) [...]
Comparando la salida de un disco sin “site” configurado con uno que sí que lo tiene, vemos que la diferencia está en los dos últimos dígitos de cada línea, que coincidirán con el siteGUID del comando “crsctl query cluster site -all” por ejemplo
Site 'site1' identified by GUID '37c347fd2c83ffb9bf127d0aca787ca9' in state 'ENABLED' [...]
también nos fijamos en que no aparece a “0” la columna antes del “;” como pasará con uno que no lo tenga configurado:
[root@pepito1 kfed]# grep siteguid prueba.dmp kfdhdb.siteguid[0]: 55 ; 0x088: 0x37 kfdhdb.siteguid[1]: 195 ; 0x089: 0xc3 kfdhdb.siteguid[2]: 71 ; 0x08a: 0x47 kfdhdb.siteguid[3]: 253 ; 0x08b: 0xfd kfdhdb.siteguid[4]: 44 ; 0x08c: 0x2c kfdhdb.siteguid[5]: 131 ; 0x08d: 0x83 kfdhdb.siteguid[6]: 255 ; 0x08e: 0xff kfdhdb.siteguid[7]: 185 ; 0x08f: 0xb9 kfdhdb.siteguid[8]: 191 ; 0x090: 0xbf kfdhdb.siteguid[9]: 18 ; 0x091: 0x12 kfdhdb.siteguid[10]: 125 ; 0x092: 0x7d kfdhdb.siteguid[11]: 10 ; 0x093: 0x0a kfdhdb.siteguid[12]: 202 ; 0x094: 0xca kfdhdb.siteguid[13]: 120 ; 0x095: 0x78 kfdhdb.siteguid[14]: 124 ; 0x096: 0x7c kfdhdb.siteguid[15]: 169 ; 0x097: 0xa9
En uno que no lo tiene configurado, debería estar:
[root@pepito1 kfed]# grep siteguid prueba2.dmp kfdhdb.siteguid[0]: 0 ; 0x088: 0x00 kfdhdb.siteguid[1]: 0 ; 0x089: 0x00 kfdhdb.siteguid[2]: 0 ; 0x08a: 0x00 kfdhdb.siteguid[3]: 0 ; 0x08b: 0x00 kfdhdb.siteguid[4]: 0 ; 0x08c: 0x00 kfdhdb.siteguid[5]: 0 ; 0x08d: 0x00 kfdhdb.siteguid[6]: 0 ; 0x08e: 0x00 kfdhdb.siteguid[7]: 0 ; 0x08f: 0x00 kfdhdb.siteguid[8]: 0 ; 0x090: 0x00 kfdhdb.siteguid[9]: 0 ; 0x091: 0x00 kfdhdb.siteguid[10]: 0 ; 0x092: 0x00 kfdhdb.siteguid[11]: 0 ; 0x093: 0x00 kfdhdb.siteguid[12]: 0 ; 0x094: 0x00 kfdhdb.siteguid[13]: 0 ; 0x095: 0x00 kfdhdb.siteguid[14]: 0 ; 0x096: 0x00 kfdhdb.siteguid[15]: 0 ; 0x097: 0x00
Nos creamos una copia de los dumps generados sustituyendo las entradas donde aparece el “siteguid” y dejamos estos ficheros nuevos en el directorio “mod”, creado anteriormente:
[root@pepito1 kfed]# for i in `ls orig/`; do awk '{ if ($1 ~ /^kfdhdb.siteguid/) printf "%-37s %s %s %s %s\n",$1,"0",$3,$4,"0x00"; else print;}' /home/grid/arumel/kfed/orig/$i > mod/$i; done
Comparamos las diferencias entre los ficheros antiguos y nuevos, con la información original y la nueva modificada:
[root@pepito1 kfed]# for i in `ls orig/`; do diff orig/$i mod/$i ;done
Paramos el clusterware en todos los nodos del cluster:
[root@pepito1 kfed]# crsctl stop crs [root@pepito2 ~]# crsctl stop crs [root@pepito3 ~]# crsctl stop crs
Deshabilitamos el AFD en uno de los nodos (es necesario para que la operación ‘write’ del kfed permita escribir sobre los discos de asm):
[root@pepito1 kfed]# asmcmd afd_filter -d
Cargamos los nuevos ficheros modificados:
[root@pepito1 kfed]# bash carga_dmps.sh Cargando el fichero /home/grid/arumel/kfed/mod/PRODATA101.dump ................ kfedDiscover: Discovering device kfedDiscover: /dev/mapper/PRODATA101p1 is a UFS disk kfedDevOpen: Opening device /dev/mapper/PRODATA101p1 from previous discovery NOTE: kfedOperation : op=write dev=/dev/mapper/PRODATA101p1 text=/home/grid/arumel/kfed/mod/PRODATA101.dump chksum=yes verb=yes (RESULT : Success) Cargando el fichero /home/grid/arumel/kfed/mod/PRODATA102.dump ................ kfedDiscover: Discovering device kfedDiscover: /dev/mapper/PRODATA102p1 is a UFS disk kfedDevOpen: Opening device /dev/mapper/PRODATA102p1 from previous discovery NOTE: kfedOperation : op=write dev=/dev/mapper/PRODATA102p1 text=/home/grid/arumel/kfed/mod/PRODATA102.dump chksum=yes verb=yes (RESULT : Success) [...]
Habilitamos el AFD en el nodo que lo habíamos deshabilitado:
[root@pepito1 kfed]# asmcmd afd_filter -e
Arrancamos el clusterware en todos los nodos del cluster:
[root@pepito1 kfed]# crsctl start crs [root@pepito2 ~]# crsctl start crs [root@pepito3 ~]# crsctl start crs
Y ya tenemos eliminados los discos con los sites!!!:
SYS@+ASM1> col path format a20 SYS@+ASM1> col failgroup format a20 SYS@+ASM1> col name format a20 SYS@+ASM1> col site_name format a20 SYS@+ASM1> select INST_ID,GROUP_NUMBER,DISK_NUMBER,MOUNT_STATUS,HEADER_STATUS,MODE_STATUS,STATE,NAME,FAILGROUP,SITE_NAME,PATH from gv$asm_disk order by 1,2,3; INST_ID GROUP_NUMBER DISK_NUMBER MOUNT_S HEADER_STATU MODE_ST STATE NAME FAILGROUP SITE_NAME PATH ---------- ------------ ----------- ------- ------------ ------- -------- -------------------- -------------------- -------------------- -------------------- [...] 1 1 0 CACHED MEMBER ONLINE NORMAL DGACFS101 FGSITE1 AFD:TESTSPLIT101 1 1 1 CACHED MEMBER ONLINE NORMAL DGACFS102 FGSITE1 AFD:TESTSPLIT102 1 1 2 CACHED MEMBER ONLINE NORMAL DGACFS201 FGSITE2 AFD:TESTSPLIT201 1 1 3 CACHED MEMBER ONLINE NORMAL DGACFS202 FGSITE2 AFD:TESTSPLIT202 1 2 0 CACHED MEMBER ONLINE NORMAL DGASM101 FGSITE1 AFD:TESTEXT101 1 2 1 CACHED MEMBER ONLINE NORMAL DGASM102 FGSITE1 AFD:TESTEXT102 1 2 4 CACHED MEMBER ONLINE NORMAL TESTEXT201 FGSITE2 AFD:TESTEXT201 1 2 5 CACHED MEMBER ONLINE NORMAL TESTEXT202 FGSITE2 AFD:TESTEXT202 1 3 0 CACHED MEMBER ONLINE NORMAL PROMGMTDB101 FGPROMGMTDB1 AFD:PROMGMTDB101 1 3 1 CACHED MEMBER ONLINE NORMAL PROMGMTDB202 FGPROMGMTDB2 AFD:PROMGMTDB202 1 3 2 CACHED MEMBER ONLINE NORMAL PROMGMTDB201 FGPROMGMTDB2 AFD:PROMGMTDB201 1 3 3 CACHED MEMBER ONLINE NORMAL PROMGMTDB102 FGPROMGMTDB1 AFD:PROMGMTDB102 1 4 0 CACHED MEMBER ONLINE NORMAL PROADMIN101 FGPROADMIN1 AFD:PROADMIN101 1 4 1 CACHED MEMBER ONLINE NORMAL PROADMIN102 FGPROADMIN1 AFD:PROADMIN102 1 4 2 CACHED MEMBER ONLINE NORMAL PROADMIN103 FGPROADMIN1 AFD:PROADMIN103 1 4 3 CACHED MEMBER ONLINE NORMAL PROADMIN104 FGPROADMIN1 AFD:PROADMIN104 1 4 4 CACHED MEMBER ONLINE NORMAL PROADMIN201 FGPROADMIN2 AFD:PROADMIN201 1 4 5 CACHED MEMBER ONLINE NORMAL PROADMIN202 FGPROADMIN2 AFD:PROADMIN202 1 4 6 CACHED MEMBER ONLINE NORMAL PROADMIN203 FGPROADMIN2 AFD:PROADMIN203 1 4 7 CACHED MEMBER ONLINE NORMAL PROADMIN204 FGPROADMIN2 AFD:PROADMIN204 1 5 0 CACHED MEMBER ONLINE NORMAL PROCTLRDO101 FGPROCTLRDO1 AFD:PROCTLRDO101 1 5 1 CACHED MEMBER ONLINE NORMAL PROCTLRDO102 FGPROCTLRDO1 AFD:PROCTLRDO102 1 5 2 CACHED MEMBER ONLINE NORMAL PROCTLRDO103 FGPROCTLRDO1 AFD:PROCTLRDO103 1 5 3 CACHED MEMBER ONLINE NORMAL PROCTLRDO104 FGPROCTLRDO1 AFD:PROCTLRDO104 1 5 4 CACHED MEMBER ONLINE NORMAL PROCTLRDO105 FGPROCTLRDO1 AFD:PROCTLRDO105 1 5 5 CACHED MEMBER ONLINE NORMAL PROCTLRDO106 FGPROCTLRDO1 AFD:PROCTLRDO106 1 5 6 CACHED MEMBER ONLINE NORMAL PROCTLRDO201 FGPROCTLRDO2 AFD:PROCTLRDO201 1 5 7 CACHED MEMBER ONLINE NORMAL PROCTLRDO202 FGPROCTLRDO2 AFD:PROCTLRDO202 1 5 8 CACHED MEMBER ONLINE NORMAL PROCTLRDO203 FGPROCTLRDO2 AFD:PROCTLRDO203 1 5 9 CACHED MEMBER ONLINE NORMAL PROCTLRDO204 FGPROCTLRDO2 AFD:PROCTLRDO204 1 5 10 CACHED MEMBER ONLINE NORMAL PROCTLRDO205 FGPROCTLRDO2 AFD:PROCTLRDO205 1 5 11 CACHED MEMBER ONLINE NORMAL PROCTLRDO206 FGPROCTLRDO2 AFD:PROCTLRDO206 1 6 0 CACHED MEMBER ONLINE NORMAL PRODATA101 FGPRODATA1 AFD:PRODATA101 1 6 1 CACHED MEMBER ONLINE NORMAL PRODATA202 FGPRODATA2 AFD:PRODATA202 1 6 2 CACHED MEMBER ONLINE NORMAL PRODATA103 FGPRODATA1 AFD:PRODATA103 1 6 3 CACHED MEMBER ONLINE NORMAL PRODATA104 FGPRODATA1 AFD:PRODATA104 1 6 4 CACHED MEMBER ONLINE NORMAL PRODATA105 FGPRODATA1 AFD:PRODATA105 1 6 5 CACHED MEMBER ONLINE NORMAL PRODATA106 FGPRODATA1 AFD:PRODATA106 1 6 6 CACHED MEMBER ONLINE NORMAL PRODATA203 FGPRODATA2 AFD:PRODATA203 1 6 7 CACHED MEMBER ONLINE NORMAL PRODATA204 FGPRODATA2 AFD:PRODATA204 1 6 8 CACHED MEMBER ONLINE NORMAL PRODATA201 FGPRODATA2 AFD:PRODATA201 1 6 9 CACHED MEMBER ONLINE NORMAL PRODATA107 FGPRODATA1 AFD:PRODATA107 1 6 10 CACHED MEMBER ONLINE NORMAL PRODATA102 FGPRODATA1 AFD:PRODATA102 1 6 11 CACHED MEMBER ONLINE NORMAL PRODATA108 FGPRODATA1 AFD:PRODATA108 1 6 12 CACHED MEMBER ONLINE NORMAL PRODATA109 FGPRODATA1 AFD:PRODATA109 1 6 13 CACHED MEMBER ONLINE NORMAL PRODATA205 FGPRODATA2 AFD:PRODATA205 1 6 14 CACHED MEMBER ONLINE NORMAL PRODATA206 FGPRODATA2 AFD:PRODATA206 1 6 15 CACHED MEMBER ONLINE NORMAL PRODATA207 FGPRODATA2 AFD:PRODATA207 1 6 16 CACHED MEMBER ONLINE NORMAL PRODATA208 FGPRODATA2 AFD:PRODATA208 1 6 17 CACHED MEMBER ONLINE NORMAL PRODATA209 FGPRODATA2 AFD:PRODATA209 1 7 0 CACHED MEMBER ONLINE NORMAL PROFRA101 FGPROFRA1 AFD:PROFRA101 1 7 1 CACHED MEMBER ONLINE NORMAL PROFRA202 FGPROFRA2 AFD:PROFRA202 1 7 2 CACHED MEMBER ONLINE NORMAL PROFRA103 FGPROFRA1 AFD:PROFRA103 1 7 3 CACHED MEMBER ONLINE NORMAL PROFRA104 FGPROFRA1 AFD:PROFRA104 1 7 4 CACHED MEMBER ONLINE NORMAL PROFRA105 FGPROFRA1 AFD:PROFRA105 1 7 5 CACHED MEMBER ONLINE NORMAL PROFRA106 FGPROFRA1 AFD:PROFRA106 1 7 6 CACHED MEMBER ONLINE NORMAL PROFRA201 FGPROFRA2 AFD:PROFRA201 1 7 7 CACHED MEMBER ONLINE NORMAL PROFRA107 FGPROFRA1 AFD:PROFRA107 1 7 8 CACHED MEMBER ONLINE NORMAL PROFRA203 FGPROFRA2 AFD:PROFRA203 1 7 9 CACHED MEMBER ONLINE NORMAL PROFRA108 FGPROFRA1 AFD:PROFRA108 1 7 10 CACHED MEMBER ONLINE NORMAL PROFRA109 FGPROFRA1 AFD:PROFRA109 1 7 11 CACHED MEMBER ONLINE NORMAL PROFRA110 FGPROFRA1 AFD:PROFRA110 1 7 12 CACHED MEMBER ONLINE NORMAL PROFRA102 FGPROFRA1 AFD:PROFRA102 1 7 13 CACHED MEMBER ONLINE NORMAL PROFRA204 FGPROFRA2 AFD:PROFRA204 1 7 14 CACHED MEMBER ONLINE NORMAL PROFRA205 FGPROFRA2 AFD:PROFRA205 1 7 15 CACHED MEMBER ONLINE NORMAL PROFRA206 FGPROFRA2 AFD:PROFRA206 1 7 16 CACHED MEMBER ONLINE NORMAL PROFRA207 FGPROFRA2 AFD:PROFRA207 1 7 17 CACHED MEMBER ONLINE NORMAL PROFRA208 FGPROFRA2 AFD:PROFRA208 1 7 18 CACHED MEMBER ONLINE NORMAL PROFRA209 FGPROFRA2 AFD:PROFRA209 1 7 19 CACHED MEMBER ONLINE NORMAL PROFRA210 FGPROFRA2 AFD:PROFRA210 1 8 0 CACHED MEMBER ONLINE NORMAL VOTOCR101 FGVOTOCR101 site1 AFD:VOTOCR101 1 8 1 CACHED MEMBER ONLINE NORMAL VOTOCR102 FGVOTOCR102 site1 AFD:VOTOCR102 1 8 2 CACHED MEMBER ONLINE NORMAL VOTOCR103 FGVOTOCR103 site1 AFD:VOTOCR103 1 8 3 CACHED MEMBER ONLINE NORMAL VOTOCR201 FGVOTOCR201 site2 AFD:VOTOCR201 1 8 4 CACHED MEMBER ONLINE NORMAL VOTOCR202 FGVOTOCR202 site2 AFD:VOTOCR202 1 8 5 CACHED MEMBER ONLINE NORMAL VOTOCR203 FGVOTOCR203 site2 AFD:VOTOCR203 1 8 6 CACHED MEMBER ONLINE NORMAL VOTOCR301 FGVOTOCR301 site3 AFD:VOTOCR301 83 rows selected.
Añadiendo valor con Arumel!!