did you pre-allocate the ISPF datasets that ship with the DB2 installation prior to invoking DSNTINST?
If not, search the names of your (DSN810.)SDSNSPFM, SDSNSPFP, SDSNSPFS, SDSNSPFT datasets, and then execute ISPF LIBDEF services for ISPMLIB, ISPPLIB, ISPSLIB, ISPTLIB. Additionally, you may need to execute an ALTLIB ACTIVATE for application(CLIST) command for the (DSN810.)SDSNCLST library.
Once done, retry calling DSNTINST.
After leaving the panel system, don't forget to free allocated libraries using LIBDEF and ALTLIB DEACTIVATE commands.
For better handling (and reuse) you may write a REXX doing the job for you. If you do write a REXX allocating and freeing the datasets, you may want to call DSNTINST using an ISPF command similar to ISPEXEC SELECT CMD(...).
Links to more information regarding LIBDEF and ALTLIB commands: ISPF LIBDEF
services, TSO/E ALTLIB
If there should arise any question related to these ISPF and TSO commands, please change to the ISPF and REXX forums.