Affecting Server - AMS01-IM
Update: from the past 30 minutes everything has been up again and stable. We'll keep the ticket open for the next upcoming hours to make sure everything remains stable also.
Thanks for your patience and sorry for the inconvenience!
--
Dear customer,
We are currently performing a critical software upgrade on 1 of our 6 core switches located in AMS01.
The reason we have to prioritize is because due to a software bug, the bgp connections to our core routers keep on dropping at irregular times>
This was reported before by a customer of ours and after further investigation we noticed this was caused by a software bug.
The upgrade is in progress as we speak and some servers might notice an interruption between 10-15 minutes at most for loading the new software.
I am sorry for the inconvenience, but we consider this a critical matter which could not wait any longer.
For questions, we are available via ticket, mail and skype.
Kind regards,
Instant Dedicated support
Affecting Other - Global Layer network
Affecting Other - Datacenter
We are suffering from a power outage since this morning and it's still ongoing.
Some customers might be offline, others might now.
We hope for your understanding and we'll try to resolve it asap.
Affecting Other - Network
Since 1:25am you might have been experiencing problems with the network.
There is an ongoing DDOS since 1:25am and our network team is already busy for several hours to debug it.
Once we have an update, we will post it here.
Some people might experience no problems and others might experience irregular time outs.
Our apologies for the inconvenience caused.
Update 04:09:
Our network team is still busy with it. Nothing more we can do from our side but wait on feedback.
We will update this ticket in the morning once we have a better understanding.
Update 06:09:
The linecard on the router crashed and therefore had to be reset. This has been done and everything is up again.
We are still waiting on feedback from our network team and Brocade whether or not it's fully resolved.
Update 06:40:
We see our traffic is restoring so I assume the issues are resolved.
Still waiting on official feedback from our network team.
Update 11:27:
Everything has been stable for the past 5-6 hours so we will close this issue.
If there are any problems, feel free to contact our support.
Affecting Other - Network
Today around 3:45am CET we experienced packet loss on our servers.
After some investigation it seemed to be several DDOS attacks.
Around 4:30am CET everything became more stable but the issue was still not fully resolved.
Our engineers were still investigating the main cause of this issue.
All servers were still online, around 20% of the servers still experienced packet loss (10-30%).
Around 8:30am CET the issue was fully resolved.
We were getting a very big UDP attack but we managed to handle it.
Any customers still experiencing problems can contact us via our ticket system.
Affecting Other - Network
We are currently experiencing packet loss on our network uplinks.
Our network team is looking into it.
Update (19:23h CET):
Network is restored again, still waiting for more info from our network team
Update (19:38h CET):
Looks like 50% of the network is still unavailable to some customers, we are still looking into the problem cause.
Update (20:02h CET):
All uplinks are restored again. If there are still problems, please mail us or create a ticket! Thanks.
We are going to send out a mailing about this issue once we have more understanding.
Affecting Other - Switch1
07:45: We are currently experiencing issues with our main Switch located in Rack #1. Around 20 customers are afected
Update:
08:15: Tech has been informed about the issue
08:30: Tech is on it's way to the datacenter
09:30: Switch has been reset, no solution
09:45: Tech is busy logging in to the switch via console to see
10:12: Spanning tree was blocking traffic coming from Core 1, currently uplink from Core2 is active and all customers are online again.
This will be investigated further by our tech team.
For now, RESOLVED.
Affecting System - Network
We are currently experiencing a small network problem.
There are customers who don't notice anything while others can't access the network completely.
We are doing all we can to find the cause and fix it asap.
Update 1 (16:30h CET):
The transit provider 'Abovenet' has been removed from the network today.
Due to bad preperations from our network provider, some problems have occured with a few IP ranges
The network team is doing their best to have it resolved asap.
Update 2 (16:56h CET):
Several ranges are becoming available again, we also notice a traffic increase on our core switching.
Not all are working yet but it shouldn't take too long anymore.
Update 3 (17:07 CET):
All ranges are working again, case closed.
Affecting Other - Full network
This afternoon a 3rd router will be added to the NIKHEF location, therefore we had to disable the Abovenet and Highwinds transit, where Global Crossing would take over. This morning Global Crossing was gone so therefore a part of our network was offline. We had to reload the router to add another route for getting the full network online again.
We had a total downtime of 1-3 minutes.
Affecting Other - Servers in suite 6
We have noticed that our switch in Suite 6 is no longer responding. A big serie of clients in the 217.148.186.x and 84.244.138.x range might experience problems.
Update: 04:30
Technician called
Update 04:52
Technician is on its way.
Update 5:53
It seems that the power of the entire rack has been gone. Currently investigating.
Update 6:07
Power is back again, servers are booting up
Update 6:12
Servers are up again with exception of 1 which was the cause of this (malfunctioning power supply). The replacement of this power supply will be done asap.
