Generally, yes. I try to work from the original source artwork whenever I can. Anytime you are ENLARGING an image, this is ABSOLUTELY NECESSARY. However when reducing the size of an image it won't matter as much and when reducing and image from non-lossy file types like GIF or PNG, you can skip working from the vector art work with no noticeable loss in quality.
Additionally, its important to note that JPG compression errors are cumulative. Even if you dont resize.. simply re saving back and forth will degrade your image quality.