[Lista ArNOG] Mala experiencia CCR1072
Ivan Chapero
info en ivanchapero.com.ar
Mar Nov 29 13:35:01 ART 2016
Mario buen dato para el que tiene pensado usarla la GE para
"retro-compatibilidad".
Pero no... en el caso nuestro se están usando los SFP+ 1 y 5.
El 29 de noviembre de 2016, 12:14, Ing. Mario Clep <
marioclep en mkesolutions.net> escribió:
> Hola Ivan,
>
> Por casualidad est
>
> as usando la interfaz ethernet del equipo? Me pasó exactamente lo mismo y
> mandé un correo al soporte de MikroTik. Me dijeron que la ether1 (la única
> ether) es sólo de management, y que está conectada a una PCI-E de 400Mbps y
> a un único núcleo del equipo.
> Me dijeron que bajo ningún punto de vista use la ether1 para algo que no
> sea management.
> ᐧ
>
> Saludos y estamos en contacto!
>
> [image: photo]
> *Ing. Mario Clep*
> CTO, MKE Solutions
> Tel: +54 358 4210029 <Tel:%20+54%20358%204210029> | info en mkesolutions.net
> | http://www.mkesolutions.net | http://www.AcademiaDeEntrenamientos.com
> <http://www.academiadeentrenamientos.com/> | Paraná 416, Río Cuarto,
> Córdoba, Argentina
> <http://facebook.com/mkesolutions> <http://twitter.com/mkesolutions>
> <http://us.linkedin.com/in/mke-solutions>
>
>
>
> 2016-11-29 11:43 GMT-03:00 Ivan Chapero <info en ivanchapero.com.ar>:
>
>> La muestra de 400mbps es para mostrar que hacia lo mismo con poco trafico
>> en cuanto al desbalanceo de carga en unos cores individuales, pero el
>> listado de uso de cpu total es sobre 1.8Gbps.
>>
>> El 29 de noviembre de 2016, 11:09, Edwin Salazar <edwin.salazar en ixp.ec>
>> escribió:
>>
>>> Jose Luis, Ivan
>>>
>>> En este caso sube esporádicamente y de forma aleatoria pero no se
>>> mantiene.
>>>
>>> El trafico que estamos pasando no es el mismo, en tu caso pasan casi
>>> 400Mbps y nosotros 3.0Gbps.
>>>
>>> Adjunto una captura adicional para que lo puedan ver.
>>>
>>>
>>> # CPU LOAD IRQ DISK
>>> 0 cpu0 0% 0% 0%
>>> 1 cpu1 0% 0% 0%
>>> 2 cpu2 0% 0% 0%
>>> 3 cpu3 1% 1% 0%
>>> 4 cpu4 3% 2% 0%
>>> 5 cpu5 4% 4% 0%
>>> 6 cpu6 10% 10% 0%
>>> 7 cpu7 6% 6% 0%
>>> 8 cpu8 1% 1% 0%
>>> 9 cpu9 1% 1% 0%
>>> 10 cpu10 0% 0% 0%
>>> 11 cpu11 0% 0% 0%
>>> 12 cpu12 1% 1% 0%
>>> 13 cpu13 17% 16% 0%
>>> 14 cpu14 32% 32% 0%
>>> 15 cpu15 3% 3% 0%
>>> 16 cpu16 2% 2% 0%
>>> 17 cpu17 4% 4% 0%
>>> 18 cpu18 0% 0% 0%
>>> 19 cpu19 0% 0% 0%
>>> 20 cpu20 32% 32% 0%
>>> 21 cpu21 11% 10% 0%
>>> 22 cpu22 7% 7% 0%
>>> 23 cpu23 7% 7% 0%
>>> 24 cpu24 0% 0% 0%
>>> 25 cpu25 0% 0% 0%
>>> 26 cpu26 0% 0% 0%
>>> 27 cpu27 7% 7% 0%
>>> 28 cpu28 5% 5% 0%
>>> 29 cpu29 10% 10% 0%
>>> 30 cpu30 10% 10% 0%
>>> 31 cpu31 7% 7% 0%
>>> 32 cpu32 18% 17% 0%
>>> 33 cpu33 1% 1% 0%
>>> 34 cpu34 4% 4% 0%
>>> 35 cpu35 14% 14% 0%
>>> 36 cpu36 21% 21% 0%
>>> 37 cpu37 4% 4% 0%
>>> 38 cpu38 10% 10% 0%
>>> 39 cpu39 16% 16% 0%
>>> 40 cpu40 9% 9% 0%
>>> 41 cpu41 18% 18% 0%
>>> 42 cpu42 12% 12% 0%
>>> 43 cpu43 19% 19% 0%
>>> 44 cpu44 18% 18% 0%
>>> 45 cpu45 19% 19% 0%
>>> 46 cpu46 17% 17% 0%
>>> 47 cpu47 24% 22% 0%
>>> 48 cpu48 9% 9% 0%
>>> 49 cpu49 19% 19% 0%
>>> 50 cpu50 13% 13% 0%
>>> 51 cpu51 19% 19% 0%
>>> 52 cpu52 22% 20% 0%
>>> 53 cpu53 15% 15% 0%
>>> 54 cpu54 24% 24% 0%
>>> 55 cpu55 25% 25% 0%
>>> 56 cpu56 6% 6% 0%
>>> 57 cpu57 18% 18% 0%
>>> 58 cpu58 9% 9% 0%
>>> 59 cpu59 9% 9% 0%
>>> 60 cpu60 22% 22% 0%
>>> 61 cpu61 21% 21% 0%
>>> 62 cpu62 20% 20% 0%
>>> 63 cpu63 15% 15% 0%
>>> 64 cpu64 10% 10% 0%
>>> 65 cpu65 12% 12% 0%
>>> 66 cpu66 16% 16% 0%
>>> 67 cpu67 16% 16% 0%
>>> 68 cpu68 15% 15% 0%
>>> 69 cpu69 18% 18% 0%
>>> 70 cpu70 22% 22% 0%
>>> 71 cpu71 0% 0% 0%
>>>
>>> Salu2.
>>>
>>>
>>> ----------------------------------------------------------------------
>>>
>>> Message: 1
>>> Date: Tue, 29 Nov 2016 10:47:43 -0300
>>> From: Ivan Chapero <info en ivanchapero.com.ar>
>>> To: lista en arnog.com.ar, Jose Luis Gaspoz <gaspozj en is.com.ar>
>>> Subject: Re: [Lista ArNOG] Mikrotik CCR1072 - Buena experiencia (Edwin
>>> Salazar)
>>> Message-ID:
>>> <CAPQhFbe2d+JT65LeGUhm6BAXotC4u76vEyTW+5b+UO58HDRBNQ en mail.gmail.com>
>>> Content-Type: text/plain; charset="utf-8"
>>>
>>> Me sorprende que con firewall OFF, tengas aun mas carga que la foto de
>>> rendimiento que subi con el mismo trafico.
>>>
>>> El 29 de noviembre de 2016, 10:17, Jose Luis Gaspoz <gaspozj en is.com.ar>
>>> escribió:
>>>
>>> Edwin:
>>>
>>> Fíjate que en tu caso esta disponiendo de pocos servicios y sin embargo
>>> tenes la CPU 10 al 65% !!!! ... eso si se mantiene un tiempo indica que
>>> no
>>> es normal en la distribución de carga de procesos.
>>>
>>> Saludos
>>>
>>> Ing. Jose Luis Gaspoz
>>> Internet Services S.A.
>>> Tel: 0342-4565118
>>> Cel: 342-5008523
>>> -----Mensaje original----- From: Edwin Salazar
>>> Sent: Tuesday, November 29, 2016 9:34 AM
>>> To: lista en arnog.com.ar
>>> Subject: [Lista ArNOG] Mikrotik CCR1072 - Buena experiencia (Edwin
>>> Salazar)
>>>
>>>
>>> Amigos,
>>>
>>> En nuestro caso tenemos una implementación con ciertas diferencias al
>>> escenario planteado pero no tenemos los resultados indicados, el router
>>> se
>>> encuentra haciendo lo siguiente:
>>>
>>> - Router de borde BGP
>>> - IPv4-only
>>> - BGP (1 carrier, GGC, NAP.EC)
>>> - ruteo estático para la red interna
>>> - Disbale Connection Tracking,
>>> - NO Firewall, Mangle, NAT, DHCP, DNS, NAT
>>> - No QoS
>>> - IP services: ssh, winbox, www
>>>
>>> [#####@Borde - BGP] > sys resource cpu print
>>> # CPU LOAD IRQ
>>> DISK
>>> 0 cpu0 0% 0% 0%
>>> 1 cpu1 1% 1% 0%
>>> 2 cpu2 2% 2% 0%
>>> 3 cpu3 0% 0% 0%
>>> 4 cpu4 0% 0% 0%
>>> 5 cpu5 1% 1% 0%
>>> 6 cpu6 0% 0% 0%
>>> 7 cpu7 0% 0% 0%
>>> 8 cpu8 16% 16% 0%
>>> 9 cpu9 4% 4% 0%
>>> 10 cpu10 65% 15%
>>> 0%
>>> 11 cpu11 0% 0%
>>> 0%
>>> 12 cpu12 0% 0%
>>> 0%
>>> 13 cpu13 0% 0%
>>> 0%
>>> 14 cpu14 4% 4%
>>> 0%
>>> 15 cpu15 0% 0%
>>> 0%
>>> 16 cpu16 1% 1%
>>> 0%
>>> 17 cpu17 0% 0%
>>> 0%
>>> 18 cpu18 1% 1%
>>> 0%
>>> 19 cpu19 4% 4%
>>> 0%
>>> 20 cpu20 0% 0%
>>> 0%
>>> 21 cpu21 0% 0%
>>> 0%
>>> 22 cpu22 0% 0%
>>> 0%
>>> 23 cpu23 1% 1%
>>> 0%
>>> 24 cpu24 2% 2%
>>> 0%
>>> 25 cpu25 4% 4%
>>> 0%
>>> 26 cpu26 0% 0%
>>> 0%
>>> 27 cpu27 2% 1%
>>> 0%
>>> 28 cpu28 3% 3%
>>> 0%
>>> 29 cpu29 0% 0%
>>> 0%
>>> 30 cpu30 1% 1%
>>> 0%
>>> 31 cpu31 1% 1%
>>> 0%
>>> 32 cpu32 5% 5%
>>> 0%
>>> 33 cpu33 1% 1%
>>> 0%
>>> 34 cpu34 0% 0%
>>> 0%
>>> 35 cpu35 2% 1%
>>> 0%
>>> 36 cpu36 0% 0%
>>> 0%
>>> 37 cpu37 2% 2%
>>> 0%
>>> 38 cpu38 1% 1%
>>> 0%
>>> 39 cpu39 3% 3%
>>> 0%
>>> 40 cpu40 0% 0%
>>> 0%
>>> 41 cpu41 1% 1%
>>> 0%
>>> 42 cpu42 13% 13%
>>> 0%
>>> 43 cpu43 7% 7%
>>> 0%
>>> 44 cpu44 0% 0%
>>> 0%
>>> 45 cpu45 2% 2%
>>> 0%
>>> 46 cpu46 0% 0%
>>> 0%
>>> 47 cpu47 4% 4%
>>> 0%
>>> 48 cpu48 0% 0%
>>> 0%
>>> 49 cpu49 7% 7%
>>> 0%
>>> 50 cpu50 0% 0%
>>> 0%
>>> 51 cpu51 12% 12%
>>> 0%
>>> 52 cpu52 4% 4%
>>> 0%
>>> 53 cpu53 2% 2%
>>> 0%
>>> 54 cpu54 0% 0%
>>> 0%
>>> 55 cpu55 0% 0%
>>> 0%
>>> 56 cpu56 11% 11%
>>> 0%
>>> 57 cpu57 0% 0%
>>> 0%
>>> 58 cpu58 17% 13%
>>> 0%
>>> 59 cpu59 2% 2%
>>> 0%
>>> 60 cpu60 2% 2%
>>> 0%
>>> 61 cpu61 1% 1%
>>> 0%
>>> 62 cpu62 0% 0%
>>> 0%
>>> 63 cpu63 1% 1%
>>> 0%
>>> 64 cpu64 5% 5%
>>> 0%
>>> 65 cpu65 2% 2%
>>> 0%
>>> 66 cpu66 2% 2%
>>> 0%
>>> 67 cpu67 1% 1%
>>> 0%
>>> 68 cpu68 0% 0%
>>> 0%
>>> 69 cpu69 0% 0%
>>> 0%
>>> 70 cpu70 6% 6%
>>> 0%
>>> 71 cpu71 0% 0%
>>> 0%
>>>
>>>
>>> [####@Borde - BGP] > inter monitor-traffic sfp6-LAN
>>> name: sfp6-LAN
>>> rx-packets-per-second: 215 812
>>> rx-bits-per-second: 908.3Mbps
>>> fp-rx-packets-per-second: 215 812
>>> fp-rx-bits-per-second: 908.3Mbps
>>> rx-drops-per-second: 0
>>> rx-errors-per-second: 0
>>> tx-packets-per-second: 264 707
>>> tx-bits-per-second: 2.0Gbps
>>> fp-tx-packets-per-second: 264 707
>>> fp-tx-bits-per-second: 2.0Gbps
>>> tx-drops-per-second: 0
>>> tx-errors-per-second: 0
>>> -- [Q quit|D dump|C-z pause]
>>>
>>> Probablemente puede ser un problema con algunos modelos, en nuestro caso
>>> tenemos buenos resultados.
>>>
>>> ??
>>>
>>> Buenas,
>>> comparto una mala experiencia que tuvimos con el primer despliegue de un
>>> CCR1072 como edge-router. La intención es evaluar con otros operadores
>>> si
>>> es reversible, así como también que sirva de antecedente para otros
>>> tentados por los 72 cores o la cantidad de SFP+.
>>>
>>> Como RouterOS es un all-in-one es muy difícil comparar experiencias de
>>> otros dado que la combinatoria de features / protocolos que pueden estar
>>> activos es enorme. Por esto aclaro primero que nada el entorno de uso
>>> del
>>> equipo (y me atrevo a decir que seria el recomendado para todo router -y
>>> solo router- de frontera):
>>>
>>> - Router de "borde" de un ISP.
>>> - IPv4-only (ni siquiera el package ipv6 enabled).
>>> - eBGP hacia dos carriers, aceptando e instalando solo default-gw.
>>> - iBGP a un segundo borde para redundancia active:active.
>>> - routing estático hacia adentro (no IGP, se podría decir que el único
>>> protocolo de routing activo es BGP).
>>> - Firewalling: sin estado (ninguna regla tiene connection-state como
>>> matcheo), sin reglas de Mangle, sin reglas de NAT,
>>> "connection-tracking=no".
>>> - No se brinda ningún servicio a la red (DNS, DHCP, NAT, etc) desde el
>>> router.
>>> - QoS: nada de nada, vacío de config el apartado queue.
>>> - Desactivado temporal por típica "a ver si es esto" en RouterOS: SNMP,
>>> todos los ip services menos winbox y ssh, logging.
>>>
>>>
>>> ## Maniobras realizadas a pedido del soporte Mikrotik que no
>>> tuvieron éxito (a pesar de que algunas rompen la necesidad o la idea de
>>> un
>>> equipo en producción original):
>>> - pasar por todas las last-version de cada rama de RouterOS (bugfix,
>>> current, RC).
>>> - disable de todas la reglas de firewall (que no hacían uso de
>>> connection-tracking).
>>> - usar varias interfaces SFP+ físicas individuales en vez de trunkear
>>> todo en vlans en unas pocas (ej 10G-WANs , 10G-CORE).
>>> - actualizar el firmware del RouterBoard.
>>>
>>> ## Detalle *MUY IMPORTANTE:*
>>> - El CCR1072 se desplegó como remplazo natural del CCR1036 que estaba (y
>>> dado esto, sigue) en producción, pensando mayormente en eliminar los
>>> LAGs
>>> Nx1G y simplificar la conexión física a los carriers que se supera el
>>> Gbps
>>> de trafico contratado.
>>> - Por esta razón, en *cuanto a config el CCR1072 es un export/import
>>> literal de la config del CCR1036. Si hacemos un diff entre uno y otro
>>> solo varían las lineas referidas a las interfaces físicas* (incluso
>>> tratamos de imitar el uso de un LAG innecesario en el CCR1072 apostando
>>> a
>>> un tema de interrupciones).
>>> - Con esto quiero remarcar que a exactamente igual config y mismo
>>> tráfico
>>> agregado (equipo de remplazo) que el CCR1036, *el uso de CPU es horrible
>>> en el CCR1072*. Mal distribuido principalmente (haciendo uso de no mas
>>> de 4 o 5 cores) y generando peaks en cores individuales, combo que
>>> reduce
>>> enormemente la escalabilidad del equipo.
>>>
>>>
>>>
>>>
>>>
>>> ?
>>>
>>>
>>>
>>>
>>> ### Analizando con profile tool, en muchas ocasiones *"unclassified"*
>>> proccess es el que se lleva gran parte de los cores que se pican
>>> intercaladamente a valores altos con respecto a la carga de tráfico:
>>>
>>>
>>> CCR1072-BORDE] > tool profile cpu=2
>>> NAME CPU USAGE
>>> firewall 2 0%
>>> networking 2 1%
>>> management 2 0%
>>> routing 2 0%
>>> idle 2 82%
>>> profiling 2 1%
>>> *unclassified 2 16% **<<< WTF!*
>>>
>>>
>>> CCR1072-BORDE] > tool profile cpu=2
>>> NAME CPU USAGE
>>> firewall 2 1%
>>> networking 2 0%
>>> management 2 0%
>>> routing 2 0%
>>> idle 2 65%
>>> profiling 2 0%
>>> *unclassified 2 34% <<< WTF!*
>>>
>>>
>>> ### Incluso en horarios de poco trafico:
>>>
>>> CCR1072-BORDE] > interface monitor-traffic sfp-sfpplus5
>>> name: sfp-sfpplus5
>>> rx-packets-per-second: 40 490
>>> rx-bits-per-second: 387.9Mbps
>>> fp-rx-packets-per-second: 40 490
>>> *fp-rx-bits-per-second: 387.9Mbps*
>>> rx-drops-per-second: 0
>>> rx-errors-per-second: 0
>>> tx-packets-per-second: 40 259
>>> tx-bits-per-second: 387.6Mbps
>>> fp-tx-packets-per-second: 40 259
>>> *fp-tx-bits-per-second: 387.6Mbps*
>>> tx-drops-per-second: 0
>>> tx-errors-per-second: 0
>>>
>>> ### Como se puede ver 60 o mas cores duermen la siesta mientras un par
>>> hacen todo el trabajo, en los 1072 reina el sindicalismo :P
>>>
>>>
>>> CCR1072-BORDE] > system resource cpu print
>>> # CPU
>>>
>>> LOAD
>>> IRQ DISK
>>> 0 cpu0
>>>
>>> 0%
>>> 0% 0%
>>> 1 cpu1
>>>
>>> 0%
>>> 0% 0%
>>> 2 cpu2
>>>
>>> 0%
>>> 0% 0%
>>> 3 cpu3
>>>
>>> 0%
>>> 0% 0%
>>> 4 cpu4
>>> *
>>> 0% 0% 0%*
>>> * 5 cpu5
>>>
>>> 37% 1% 0%*
>>> 6 cpu6
>>>
>>> 0%
>>> 0% 0%
>>> 7 cpu7
>>>
>>> 0%
>>> 0% 0%
>>> 8 cpu8
>>>
>>> 0%
>>> 0% 0%
>>> 9 cpu9
>>>
>>> 0%
>>> 0% 0%
>>> 10 cpu10
>>>
>>> 0%
>>> 0% 0%
>>> 11 cpu11
>>>
>>> 0%
>>> 0% 0%
>>> 12 cpu12
>>>
>>> 0%
>>> 0% 0%
>>> 13 cpu13
>>>
>>> 0%
>>> 0% 0%
>>> 14 cpu14
>>>
>>> 0%
>>> 0% 0%
>>> 15 cpu15
>>>
>>> 0%
>>> 0% 0%
>>> 16 cpu16
>>>
>>> 0%
>>> 0% 0%
>>> 17 cpu17
>>>
>>> 6%
>>> 6% 0%
>>> 18 cpu18
>>>
>>> 0%
>>> 0% 0%
>>> 19 cpu19
>>>
>>> 0%
>>> 0% 0%
>>> 20 cpu20
>>>
>>> 0%
>>> 0% 0%
>>> 21 cpu21
>>>
>>> 0%
>>> 0% 0%
>>> 22 cpu22
>>>
>>> 0%
>>> 0% 0%
>>> 23 cpu23
>>>
>>> 0%
>>> 0% 0%
>>> 24 cpu24
>>>
>>> 0%
>>> 0% 0%
>>> 25 cpu25
>>>
>>> 0%
>>> 0% 0%
>>> 26 cpu26
>>>
>>> 0%
>>> 0% 0%
>>> 27 cpu27
>>>
>>> 0%
>>> 0% 0%
>>> 28 cpu28
>>>
>>> 0%
>>> 0% 0%
>>> 29 cpu29
>>>
>>> 0%
>>> 0% 0%
>>> 30 cpu30
>>>
>>> 0%
>>> 0% 0%
>>> 31 cpu31
>>>
>>> 0%
>>> 0% 0%
>>> 32 cpu32
>>>
>>> 0%
>>> 0% 0%
>>> 33 cpu33
>>>
>>> 0%
>>> 0% 0%
>>> 34 cpu34
>>>
>>> 0%
>>> 0% 0%
>>> 35 cpu35
>>>
>>> 2%
>>> 2% 0%
>>> 36 cpu36
>>>
>>> 0%
>>> 0% 0%
>>> *37 cpu37
>>>
>>> 34%
>>> 33% 0%*
>>> 38 cpu38
>>>
>>> 0%
>>> 0% 0%
>>> 39 cpu39
>>>
>>> 2%
>>> 2% 0%
>>> 40 cpu40
>>>
>>> 0%
>>> 0% 0%
>>> 41 cpu41
>>>
>>> 0%
>>> 0% 0%
>>> 42 cpu42
>>>
>>> 0%
>>> 0% 0%
>>> 43 cpu43
>>>
>>> 0%
>>> 0% 0%
>>> 44 cpu44
>>>
>>> 0%
>>> 0% 0%
>>> 45 cpu45
>>>
>>> 0%
>>> 0% 0%
>>> 46 cpu46
>>>
>>> 0%
>>> 0% 0%
>>> 47 cpu47
>>>
>>> 1%
>>> 1% 0%
>>> 48 cpu48
>>>
>>> 0%
>>> 0% 0%
>>> 49 cpu49
>>>
>>> 0%
>>> 0% 0%
>>> 50 cpu50
>>>
>>> 0%
>>> 0% 0%
>>> 51 cpu51
>>>
>>> 0%
>>> 0% 0%
>>> 52 cpu52
>>>
>>> 0%
>>> 0% 0%
>>> 53 cpu53
>>>
>>> 0%
>>> 0% 0%
>>> 54 cpu54
>>>
>>> 0%
>>> 0% 0%
>>> 55 cpu55
>>>
>>> 0%
>>> 0% 0%
>>> 56 cpu56
>>>
>>> 0%
>>> 0% 0%
>>> 57 cpu57
>>>
>>> 0%
>>> 0% 0%
>>> 58 cpu58
>>>
>>> 0%
>>> 0% 0%
>>> 59 cpu59
>>>
>>> 0%
>>> 0% 0%
>>> 60 cpu60
>>>
>>> 0%
>>> 0% 0%
>>> 61 cpu61
>>>
>>> 0%
>>> 0% 0%
>>> 62 cpu62
>>>
>>> 0%
>>> 0% 0%
>>> 63 cpu63
>>>
>>> 0%
>>> 0% 0%
>>> 64 cpu64
>>>
>>> 3%
>>> 2% 0%
>>> 65 cpu65
>>>
>>> 0%
>>> 0% 0%
>>> 66 cpu66
>>>
>>> 1%
>>> 0% 0%
>>> 67 cpu67
>>>
>>> 0%
>>> 0% 0%
>>> 68 cpu68
>>>
>>> 0%
>>> 0% 0%
>>> 69 cpu69
>>>
>>> 0%
>>> 0% 0%
>>> 70 cpu70
>>>
>>> 0%
>>> 0% 0%
>>> 71 cpu71
>>>
>>> 0%
>>> 0% 0%
>>>
>>>
>>> --
>>>
>>> *Ivan ChaperoÁrea Técnica y Soporte*
>>> Fijo: 03464-470280 (interno 535) | Móvil: 03464-155-20282 | Skype ID:
>>> ivanchapero
>>> --
>>> GoDATA Banda Ancha - CABLETEL S.A. | Av. 9 de Julio 1163 - 2183 -
>>> Arequito - Santa Fe - Argentina
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Lista mailing list
>>> Lista en arnog.com.ar
>>> http://mailmancabase.interdotnet.com.ar/mailman/listinfo/lista
>>>
>>>
>>>
>>> _______________________________________________
>>> Lista mailing list
>>> Lista en arnog.com.ar
>>> http://mailmancabase.interdotnet.com.ar/mailman/listinfo/lista
>>>
>>>
>>> _______________________________________________
>>> Lista mailing list
>>> Lista en arnog.com.ar
>>> http://mailmancabase.interdotnet.com.ar/mailman/listinfo/lista
>>>
>>>
>>> ------------ próxima parte ------------
>>>
>>> Se ha borrado un adjunto en formato HTML...
>>> URL: <http://mailmancabase.interdotnet.com.ar/pipermail/lista/
>>> attachments/20161127/86499f69/attachment.html>
>>>
>>> ------------------------------
>>>
>>> _______________________________________________
>>> Lista mailing list
>>> Lista en arnog.com.ar
>>> http://mailmancabase.interdotnet.com.ar/mailman/listinfo/lista
>>>
>>>
>>> Fin de Resumen de Lista, Vol 114, Envío 8
>>> *****************************************
>>>
>>>
>>> _______________________________________________
>>> Lista mailing list
>>> Lista en arnog.com.ar
>>> http://mailmancabase.interdotnet.com.ar/mailman/listinfo/lista
>>>
>>>
>>>
>>>
>>> -----
>>> Se certificó que el correo no contiene virus.
>>> Comprobada por AVG - www.avg.com
>>> Versión: 2016.0.7924 / Base de datos de virus: 4728/13500 - Fecha de la
>>> versión: 29/11/2016
>>>
>>>
>>>
>>> _______________________________________________
>>> Lista mailing list
>>> Lista en arnog.com.ar
>>> http://mailmancabase.interdotnet.com.ar/mailman/listinfo/lista
>>>
>>>
>>>
>>>
>>> --
>>>
>>> *Ivan ChaperoÁrea Técnica y Soporte*
>>> Fijo: 03464-470280 (interno 535) | Móvil: 03464-155-20282 | Skype ID:
>>> ivanchapero
>>> --
>>> GoDATA Banda Ancha - CABLETEL S.A. | Av. 9 de Julio 1163 - 2183 -
>>> Arequito
>>> - Santa Fe - Argentina
>>> ------------ próxima parte ------------
>>> Se ha borrado un adjunto en formato HTML...
>>> URL: <http://mailmancabase.interdotnet.com.ar/pipermail/lista/att
>>> achments/20161129/2b99e494/attachment.html>
>>>
>>> ------------------------------
>>>
>>> _______________________________________________
>>> Lista mailing list
>>> Lista en arnog.com.ar
>>> http://mailmancabase.interdotnet.com.ar/mailman/listinfo/lista
>>>
>>>
>>> Fin de Resumen de Lista, Vol 115, Envío 2
>>> *****************************************
>>>
>>>
>>>
>>> _______________________________________________
>>> Lista mailing list
>>> Lista en arnog.com.ar
>>> http://mailmancabase.interdotnet.com.ar/mailman/listinfo/lista
>>>
>>>
>>
>>
>> --
>>
>> *Ivan ChaperoÁrea Técnica y Soporte*
>> Fijo: 03464-470280 (interno 535) | Móvil: 03464-155-20282 | Skype ID:
>> ivanchapero
>> --
>> GoDATA Banda Ancha - CABLETEL S.A. | Av. 9 de Julio 1163 - 2183 -
>> Arequito - Santa Fe - Argentina
>>
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> Lista mailing list
>> Lista en arnog.com.ar
>> http://mailmancabase.interdotnet.com.ar/mailman/listinfo/lista
>>
>>
>
> _______________________________________________
> Lista mailing list
> Lista en arnog.com.ar
> http://mailmancabase.interdotnet.com.ar/mailman/listinfo/lista
>
>
--
*Ivan ChaperoÁrea Técnica y Soporte*
Fijo: 03464-470280 (interno 535) | Móvil: 03464-155-20282 | Skype ID:
ivanchapero
--
GoDATA Banda Ancha - CABLETEL S.A. | Av. 9 de Julio 1163 - 2183 - Arequito
- Santa Fe - Argentina
------------ próxima parte ------------
Se ha borrado un adjunto en formato HTML...
URL: <http://mailmancabase.interdotnet.com.ar/pipermail/lista/attachments/20161129/e77493a7/attachment-0001.html>
------------ próxima parte ------------
A non-text attachment was scrubbed...
Name: Captura de pantalla 2016-11-29 a las 08.57.05.png
Type: image/png
Size: 119554 bytes
Desc: no disponible
URL: <http://mailmancabase.interdotnet.com.ar/pipermail/lista/attachments/20161129/e77493a7/attachment-0001.png>
Más información sobre la lista de distribución Lista