Peers keep contacting me

From AzureusWiki

With any BitTorrent client, you are connected to at least one tracker (either a central tracker or a distributed tracking system). The tracker stores your IP address and port.

Other clients can obtain your IP address and port from the tracker, but may not use it right away. Your client will tell the tracker when you log off, but other clients may try to contact you before being notified by the tracker.

It is possible that they have been told by the tracker that you are no longer connected, but due to a buggy or rogue client they are ignoring this.

Providing the peer has a correct client, it will stop the contacts when the tracker lets everyone know you have gone, or when the remote peer gives up after several minutes. This may happen for 5 to 15 minutes.

Read the Azureus FAQ