When polling a JMS queue in OSB, in order for it to go to the configured error queue, an error must be raised. Raising an error and then handling this error won't cause the message to be sent to the error queue. If the error is handled, then a fault must be raised again.
Take this simple proxy service as an example that polls from a JMS queue configured in weblogic and calls a stored procedure to insert the polled record:
EmployeePS Pipeline:
Should an unexpected error occur somewhere in this pipeline, it will be caught by the error handler. The error will be reported in the EM Message Reports but the error is raised again so that it can be sent to the error queue.
Doing a Reply with Failure will NOT send the message to the error queue. As explained by AbhishekJ here:
In case of JMS transport the JMS Queue or Topic will receive a NACK back but the message will NOT roll back and will be removed from the destination just like in case of Reply with Success. This happens because using a Reply action is considered as an exception being handled properly and a commit will be sent back to the JMS destination. To roll back the message ensure that you use Raise Error instead of a Reply action.
See also:
Transaction handling within OSB
Transaction handling within OSB