Our boundary fence was attached in 2006 to our now deceased

Our boundary fence was attached in 2006 to our now deceased neighbors shed wall with his permission. The house has now been sold just before Xmas 2013. We had to tell them that our fence was attached to the shed wall when he started to demolish it a week or so ago. He left the wall there after demolishing the rest of the shed. Unsupported. The weather has blown it down taking our fence with it. He is telling us that he has been told by the council that he is not responsible for the fence as it was the wind!! Even though he left it unstable. He is going to repair it he says as he is being a good neighbor!! He is also saying that we shouldn't have fixed it to his shed.  It wasn't his when the fence was fixed to the shed. Would a servey show this sort of thing up. We want to know if this is our responsibility or is he responsible because he demolished all but one wall of the shed making it weak and unsafe?  He is getting a bit nasty because we want it done now and he is putting it off because he says the weather is bad. Many thanks Lorraine