At most fishable flows, the NB of the Potomac will eat a drift boat. If you run a plastic (hdpe) boat, it might be doable, but I would be really cautious. There is a reason you see mostly rafts floating down there.
If you are a warmwater guy, you have plenty of options within a few hours of driving. Focusing on trout limits you a lot more with regard to locations and even flow conditions.
I run a Clacka here in SWPA and the floating for trout thing can get pretty bumpy here by June.
A wilderness float?