Why does the paperclip icon not disappear when I delete my attachment?

If an attachment (e.g. to a sales order or customer account) is removed, then the paperclip icon may remain for a while until the server updates, but the attachment has indeed been removed.