Returning greencard

[immihelp]

To whom it may concern,

I became a citizen of the United States last year. Recently I travel outside the U.S. and when I came back I was hold by immigration because my records show that I never returned my permanent resident card. When I applied and took my oath to become a citizen no one ask me to return the card. My question is where do I have to return this item so I don't go get in trouble every time I come back from vacation.
Thanks,

[immihelp]

You were supposed to return it at the time of oath.