I think the court does actually take into account why you refuse to answer. If it's "because I was waiting for my lawyer to arrive, and then I answered the ques

I think the court does actually take into account why you refuse to answer. If it's "because I was waiting for my lawyer to arrive, and then I answered the question just fine", I'm not sure a court could rightly draw an adverse inference there. The "we can infer guilt" seems to be more because there's a bunch of corner cases where an absolute right to silence CAN have kind of stupid outcomes. Many countries choose to accept those outcomes because they don't want to convict the innocent. Britain -otoh- seems to try to thread the needle here by giving the judge a bit of leeway. But I'm not an expert on british law. I just online-searched the the heck out of it.