Tired of playing with same set of effects that come with Photo Booth? Well, after poking around, I came across an article that (sorta) explains how to do it. So, I took the best parts, cut out the stuff you don't care about and jotted the steps down here for you:
1. In Finder, navigate to /System/Library/Compositions
2. Use command-click to select all the recommended compositions: ASCII Art, Blue Print, Blur, City Lights, Color Controls, Color Invert, Compound Eye, Concert, Crystallize, Dot Screen, Exposure Adjust, Gamma Adjust, Kaleidescope, Line Overlay, Line Screen, Monochrome, Neon, Pixellate, Pointillize, Posterize, Sharpen, Tracer, Zoom Blur
3. Copy the items
4. In Finder, navigate to MacintoshHD/Users/your_name/Library/Compositions (if that folder does not exist on your computer, then you can just create it). If you are familiar with the terminology, you can navigate to ~/Library/Compositions.
5. Paste the items
6. Open a Terminal window, and execute the following commands:
for i in *.qtz; do /usr/libexec/PlistBuddy -c 'delete :excludedHosts:' "$i"; done
(for BEST results, just copy the line above and paste it into your Terminal session, then press ENTER/RETURN)
**Ignore any "errors" you might see during the execution of that command.
7. Launch PhotoBooth, look for the new effects at the right of your effects list and enjoy!
This information can be found here: http://www.macworld.com/article/143700/2009/11/photoeffects.html?lsrc=nl_mwhints_h_crawl