I want to set up some basic REST services for my XPage application. So I added the xe:restService control on an xpage and choose the xe:customRestService where I refer to a Java class:

<xe:restService id="restService1" pathInfo="json" state="false">
        <xe:this.service>
            <xe:customRestService contentType="application/json"
                serviceBean="se.banking.desk.CustomSearchHelper">
            </xe:customRestService>
        </xe:this.service>
    </xe:restService>

The CustomSearchHelper class it self is still pretty empty but I am wondering if I am on the right track?

Here is the code for the class:

package se.banking.desk;

import java.io.IOException;
import java.util.Map;

import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import com.ibm.domino.services.ServiceException;
import com.ibm.domino.services.rest.RestServiceEngine;
import com.ibm.xsp.extlib.component.rest.CustomService;
import com.ibm.xsp.extlib.component.rest.CustomServiceBean;

public class CustomSearchHelper extends CustomServiceBean {

    @Override
    public void renderService(CustomService service, RestServiceEngine engine) throws ServiceException {

        HttpServletRequest request = engine.getHttpRequest();           
        String method = request.getMethod();

        HttpServletResponse response = engine.getHttpResponse();        
        response.setHeader("Content-Type", "text/javascript; charset=UTF-8");        

        if(method.equals("GET")){
            this.get(engine);
        }
        else if(method.equals("POST")){
            this.post(engine,request);
        }
        else{
            this.other(engine);
        }

    }

    public void get(RestServiceEngine engine){
        HttpServletResponse response = engine.getHttpResponse();
        try {
            response.getWriter().write("get()");
            response.getWriter().close();
            return;
        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
    }

    public void post(RestServiceEngine engine,HttpServletRequest request){
      HttpServletResponse response = engine.getHttpResponse();
      Map parameters = request.getParameterMap();
      try {
          response.getWriter().write("post()");
          response.getWriter().write( request.getParameter("form"));
          String[] form = (String[])parameters.get("form");
          String val = form[0];
          response.getWriter().write(val);
        response.getWriter().close();
    } catch (Exception e) {
        // TODO: handle exception
    }     

  }

    public void other(RestServiceEngine engine){
       HttpServletResponse response = engine.getHttpResponse();
        try {
            response.getWriter().write("other()");
            response.getWriter().close();
            return;
        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
    }

}

Questions I have: Is this a good way to write a custom REST service? Are there alternatives? Where can I find more examples/information that start at the starter level?

You've asked a rather complex question, one which has been on my mind for the last couple of years. My assessment is that finding the "good way" comes down to the developer(s) and conventions used within the app. I've included links to sources of the alternatives I see available, a few mine, those of which attempted to tackle some of the concepts from the ground up, like my series on http servlets.

[Update]I've edited this answer to include some code examples, as there is always the possibility of links eventually not working; which should preserve the intent of the answer.[/Update]

Your implementation is a great example of how the xe:restService control can be easily bound to an XPage, with the wide variety of options available to use within the XPages runtime and Domino server.

So far as I can tell, there are about 5 unique(-ish) ways of implementing a RESTful API/endpoint for operation within an XPages context. In general order of ease of implementation (depending on the person):

1. XAgent (this could be swapped with the next one; benefits in being easy to start with, lots of examples available, and less daunting to those not experienced with Java, can be done in SSJS or Java)

<?xml version="1.0" encoding="UTF-8"?>
<xp:view
    xmlns:xp="http://www.ibm.com/xsp/core"
    rendered="false"
    viewState="nostate">
    <xp:this.afterRenderResponse>
        <![CDATA[#{javascript:com.demo.DataProvider.myCustomDataServiceAsJson();}]]>
    </xp:this.afterRenderResponse>

    XAgent. This will not render as a page, but as application/json data.
</xp:view>

2. the best part about the xe:restService is the out-of-the-box options for things like xe:viewJsonService (Domino devs already think in terms of Views and Documents, which are somewhat analogous to a RESTful API's mechanics of collections and records), as Mark pointed out in the comments; these are rather easy to get into and convenient (many people have blogged about these, Brad has covered some of these rather extensively in his data grids series)
3. xe:restService, either as a CustomServiceBean as you've shown above