First off is that openssl command correct? Should it be this instead:
openssl pkcs8 -inform der -nocrypt -in test.der -out result.pem
Try out your openssl command within a command prompt so that you know that it works ok. I think the command line you specified waits on stdin (well it does for me).
runtime.exec creates a Process object. If you do this:
Process openssl = runtime.exec("....")
then you can examine the return code from openssl to see the exit code - for instance if the input file does not exist then exit = 1. You can test for this with Java
Alternatively you could get the stderr from the process and look inside it - if it is 0 length then all is good, if it has some text in there then it has likely failed. You could then throw an exception and include the stderr output in the exception messgae. You may need to experiment with this, runnig it first when openssl is happy then running it again when openssl is upset.