If I were to purchase a fan package, would only I would be able to get into the panel?

Panels are open to everyone (space-permitting). It’s the premium seating that would be only available to the individual who purchased the fan package. If you opted to not take advantage of the premium seating available with the Fan Package, that isn’t a problem and you could sit with the other members of your party (just without the premium view!).