The documentation covers the similar case of purging a file from history:

git filter-branch --index-filter 'git rm --cached --ignore-unmatch filename' HEAD

Since you are deleting a whole directory, add the -r flag to git rm:

git filter-branch --index-filter \
                  'git rm -r --cached --ignore-unmatch path/to/directory' HEAD

Note that this operation will take several minutes on larger repositories.

More importantly, it will make a new repository with distinct history and checksums. If you previously published your repository, the history of the new one will not be compatible with the history others have pulled.