find -depth -type d -empty -exec rmdir {} ;

The -depth parameter tells find to start from the innermost directory first and work its way to the top level.