Try the following steps:
- Determine how serious the idea should be.
1.1. If it's intended as a thinking toy, goto 2.
1.2. If it's intended to be used commertially, or even just somewhat widely, goto 3.
- Have fun, share it, etc.
2.1. Write some code to work it.
2.2. Go to your favorite puzzle/game/riddle forum and post it.
2.3. Publish your code on GitHub or similar places. And remember to put a disclaimer in your README file that it's intended for fun.
- Evaluate.
3.1. Test the efficiency of your algorithm, because most other commercially deployed algorithm can achieve high efficiency while hardly sacrificing security. If you spend an order of magnitude of more time encrypting, decrypting, signing, verifying some data than other algorithms, then stop there - it can't compete with the other algorithms.
3.2. Improve the quality of the specification (algorithm description) and the implementation (your code). If despite the best of your attempt, they look ugly, then stop there - it's unmaintanable and will most likely break some time soon.
3.3. Go to your preferred academic search engine to look up if parts of your algorithm had been analyzed, proposed, or had ever came up before. If they've not been proposed in the last 20-30 years, then there might be a reason it's not proposed, and could be an indication that it has flaws. If these all passed, goto step 4.
- Formally publish your idea.
4.1. Write a paper about your algorithm, make a reference implementation. It's okay if someone post a partial or full break of your algorithm, these types of paper comes up all the time, and it's exactly these papers that establishes the confidence in the algorithms we use.
4.2. See if there's a cryptography standardization project opening. Submit your algorithm according to their rules; discuss your algorithm at project (tele)conferences; and hope it makes it to the end.