Are you very, very sure that you've set the TX to serializable? Because that should never happen on a serializable transaction.

If two TX read and modify the same row in a serializable transaction, then oracle throws an ORA-08177.

Please check that hibernate is actually setting the TX as serializable.

Edit

You can do what Jonas suggested or you can also check it from you application by getting the underlying connection and invoking Connection.getIsolationLevel(). For example

Connection c = session.connection()
int level = c.getIsolationLevel()

Edit 2

Ok, since you confirmed that the isolation level on the Connection is SERIALILIZABLE, could you check:

• That the tables are using the innoDB engine.
• As Jonas suggested, run SELECT @@tx_isolation; from your code while it is in the transaction. It should return SERIALIZABLE. This is to check that the Connection is actually propagating the isolation level. It's a bit paranoid, but what to do...
• Check that your code only opens one transaction, and runs everything in that TX. I just tested the SERIALIZABLE isolation level manually and it works as expected (it blocks any TX trying to read the same row).
• Last resort: check that the isolation level SERIALIZABLE works on your MySQL installation.

NOTE: As I mentioned before, MySQL will block any queries trying to read from the same row. That means that if you have some "common tables" such as country, company, user, etc. that many TXs read concurrently, it might make your app run almost sequencially rather than parallelly.