Our FAQ currently says:
Do we accept questions asking for cryptanalysis of your cipher (hash function, ...) design?
No, we do not. If you want peer review of your full cryptographic scheme, here is not the place to acquire it. However, you might like to break your problem down into specifics, such as "under these conditions, does structure X have desired security property Y?" which would be a perfect fit for us.
As an example, this recent question was closed as off topic by Paŭlo Ebermann, citing this FAQ entry. So the answer to "Do we allow these questions?" is clearly "No." What I'd like to ask is whether we, perhaps, should allow at least some of them after all.
(I do realize I may be asking for downvotes by bringing up what may have been an issue already debated to death before — although, in my defense, if such extensive debate does exist, I didn't find it.)
I think there are some reasons why allowing questions of the type "Here's a cipher I made up, how do I break it?" here could be a good idea — or at least not a terribly bad one:
To me, questions like these would seem an excellent fit to the StackExchange Q&A format, if there only was a site that accepted them. At the moment, there doesn't seem to be one: some have been asked on both SO and math.SE, but they're not a very good fit to either; for example, the same question mentioned above was also closed as off topic on SO.
(On the other hand, this not so dissimilar question seems to have done just fine on math.SE — and was recommended as "perfect for the Cryptography.SE site" there, before this site entered public beta.)
A final point to note about this is that, whatever location is to be chosen as most appropriate for these questions, if they are to have any chance of receiving high quality answers it really ought to be one that is frequented by experts in cryptanalysis. As such, one particular StackExchange site suggests itself to me...
One concern about these questions seems to be that they'll be too open-ended to have a definite answer. However, I'd expect most such questions to have a very simple and definite answer: "Your scheme is not secure, because..." It's true that someone might occasionally come up with a cipher that stumps even the bright minds here; but such situations, as long as they remain occasional, are normal on any Q&A site, and in those cases we can at least point the asker towards further resources on how to analyze their cipher.
(Of course, we shouldn't create the impression that a homebrew cipher is secure just because none of us can break it — but I don't really believe that to be a risk. I'd expect most of the regulars here to instinctively include such disclaimers in their answers anyway, whenever the situation might possibly call for them.)
Another concern may have been that such questions might become too common and crowd out more relevant questions. My suggestions would be not to worry about that before it happens. (I wanted to say "not to count our chickens until they've hatched", but that doesn't really have the right connotations... perhaps I should ask on english.SE for a better idiom.) Anyway, we can always revisit the issue if and when such questions do become too common.
(My experience on the completely unmoderated Usenet forum sci.crypt suggests that they probably won't; they do appear from time to time, but not often enough to crowd out other topics. The threads that drown out meaningful discussion on sci.crypt are generally completely different ones.)
Finally, our FAQ also says that "Crypto - Stack Exchange is for software developers, mathematicians and others interested in cryptography" (emphasis mine). I believe this should also include amateur cryptographers interested in developing and analyzing new crypto schemes, whether to practice their cipher design and cryptanalytic skills or just because they like such mental challenges. After all, some of these people will become the next generation of crypto experts.
To recap, I'd suggest amending our FAQ (and practice) to explicitly allow questions of the form "I made up this cipher/hash/etc., how can I break it?" or "...what are its weaknesses?".
(The common phrasing "... is it secure?" is somewhat unfortunate, but I do believe that the underlying questions could still often be made appropriate to this site. In any case, the literal questions do generally have a definite answer: either "No, because..." or "Probably not, but we can't tell", with the very occasional "Yes, assuming..." thrown in.)