As long as your doctor is only prescribing long-acting narcotics for people with clinical evidence of severe pain, and s/he utilizes opioid contracts, I wouldn't worry overmuch.
When my PCP was asked about my methadone prescriptions (prior to the pain management doctor taking over my care) he simply had to supply medical evidence of need and they left him alone.
In my case, MRI results and 12 years of successive failed treatment options back up my need for long-acting opioids.
It sounds like you have substantial medical evidence of need and so I wouldn't be too concerned.
It does seem odd that your PM doc would not only tell you about the letter, but show it to you as well. I wonder why he did that?
-------------------- Never underestimate the predictability of stupidity.