Does it matter one way or the other whether I have 'follow' or 'nofollow' set up in my comments configurations, in terms of the seo of my site?
you can set it to dofollow,and specify that only if the comment are well written,they will be published,then you will get some quality fresh content everyday,and don't forget to index the comment
If you get comments from a higher page rank site then you will have the chance of flowing that pages value to your page and eventually increase your PageRank. This is according to Google's engineer Mattcutt's. So I would suggest, if you can edit each comment made by your commenter you can identify which one to allow on your page.
You can always engage in "linksculpting" where you set internal pages and valuable external pages to follow and less valuable internal/external pages to no follow links.
You can do this using the old robots.txt in the HTML
Set to nofollow will be better for your site in terms of SEO. Because that will be leading less links out from your site. That's why most of sites set their link type to be 'nofollow'.
if you set your blogs as no follow, you'll receive less spam and if its set as do follow, expect for more spammy comments..