You might consider avoiding this issue altogether by writing a different loop.

Consider using the gen.factorial function in AlgDesign, a la:

fact1 = gen.factorial(c(length(i.vector), length(j.vector), length(k.vector)), nVars = 3, center = FALSE)
foreach(ix_row = 1:nrow(fact1)) %dopar% {
  my.function(fact1[ix_row,])
}

You could also use memory mapped files and pre-allocate the output storage using bigmemory (assuming you're creating a matrix) and that would make it feasible for each worker to store its output on its own.

In this way, your overall memory usage should drop dramatically.

Update 1: It seems that memory issues are endemic to doSMP. Check out the following posts:

• Answer by Revo engineer discusses some memory & process issues
• Joris Meys reports that doSMP crashes his R instances frequently

I recall seeing another memory issue for doSMP, either on as a question or in the R chat, but I can't seem to recover the post.

Update 2: I don't know if this will help, but you might try using an explicit return() (e.g. return(my.function(ith.value, jth.value, kth.value))). In my code, I generally use an explicit return() for clarity.