How do I gain enough knowledge to experiment with cryptography?

Question

There seems to be a common trend among forums involving the topic of cryptography to shutdown any conversation very quickly, least in my experience.

From the stand point of a newcomer to the field making bold claims and selling snake oil I can understand the trepidation and warnings. But as someone who has been reading the books, studying the courses, and experimenting with code when do they graduate to being able to ask educated questions and get more discourse beyond “thou shall not write your own stuff?”

For example. I know enough to not write my own cipher. I use battle tested technology like AES-GCM. But what if my experiment wishes to use it in a different way. Maybe a communication protocol for a learning exercise? Or chunking algorithm that uses the cipher for larger payloads?

When addressing these application level ideas I find it very hard to get any dialogue or information because I’m not allowed to do it myself even if the system I’m targeting lacks something.

My deepest apologies if this is off topic to this SE. I’m just at a loss on how to find educational discourse about cryptography and its use without the gate keeping that happens in this community.

Answer 1

CAVEAT: I'm not sure how to answer this question other than with opinions. Other opinions exist and polite/constructive debate is welcomed. I also reserve the right to edit and update as additional thoughts occur to me.

1. Start by breaking things

Before setting out to design cryptography, whether primitives, protocols or applications, it is vital to have a good knowledge of the subtle, complex, and surprising ways in which cryptography can fail. Has anybody tried to address the design challenge that you hope to solve before? Was their attempt considered unsuccessful? If so why? Look at designs similar to what you intended to do. Are there historical attacks that have been fixed? Can you spot issues with the design? Learn the taxonomies of attacks: distinguishers that exist for primitives; replay and reflection attacks for protocols; implementation errors for products. If you can develop a nose for quickly identifying these risks, you will find it easier to avoid in your own designs.

Showing that existing cryptography is broken is always interesting to the community and will attract a ready and willing audience. This will in turn increase the population of cryptographers willing to give time and effort to analyse your own ideas.

2. Move on to fixing things

As your cryptanalytic skills develop and you become adept at finding issues with existing proposals, consider also if there is a simple fix. This activity again does significant good within the community. It is usually easier to repair than to build from scratch. The focus on modifying a part of a design limits allows you to work with a smaller scope and will provide easier tasks than blank slate work.

Working to fix designs will lead to constructive conversations with designers who can give you insights into their rationales and choices, which will help you to make your own. Again, it widens the group who will devote time and effort to considering your personal design work.

3. Try to work with a team

As mentioned, cryptography can go wrong in many subtle and unexpected ways. Having many eyes on a design project at all stages will improve the chances of spotting issues early. All cryptographers have both blindspots and issues that seem obvious to them. A team can overlap and provide a healthier overall level of coverage.

4. Design something that meets a need

Inevitably, you will need to learn cryptographic design by making mistakes and learning from them. You will need knowledgeable and experienced cryptanalysts to identify and explain these mistakes to you. Knowledgeable and experienced cryptanalysts are deluged with requests to analyse designs from a very wide spectrum of design expertise (crypto.SE has a specific policy of refusing to analyse designs). To benefit from the expertise of these people, it helps if they believe you've made a good start on the cryptologic journey (see step 1 and 2), but also that the work is of benefit to the community. If there are existing designs that fulfil similar functionality to your design, then what does you design bring to the table? Is it simpler? Does it use less/fewer resources? Does it provide new security features? If not, the work is unlikely to be used and professionals will be less willing to give time to its analysis.

5. Check your own homework but get someone else to mark it

It goes without saying that during the design process, you should regularly scrutinise your ideas with an adversarial eye. How would you as an attacker seek to compromise the security? Are there tacit assumptions that you have made that an attacker would be able to negate or ignore? However, once you do have a finalised design, you should seek out someone not involved in the design process who will be seeing it for the first time and ask them to do a thorough review.

It is important to be as specific as possible when handing over the final design. If designing a primitive, provide test vectors and intermediate computations. If designing a protocol, be specific in the choice of primitives and modes that you would like to use as well as being exhaustive in describing how the protocol deals with failures at various steps. If providing an implementation, make sure that the API is well-documented. It is incredibly frustrating as a cryptanalyst to have to deal with an ambiguous specification, or to have a designer claim that an attack is not valid due to an omission of that case in the specification.

When your reviewer does come back with comments and attacks, be thankful and humble. Ask them how they came to consider their approach and if it can be mitigated. If you feel that elements of your design mitigate the proposed attack, ask respectfully whether they might help. Try to make your reviewer happy that they helped you rather than getting into an argument; you may need their help again.

6. Know when to iterate and when to abandon

Sooner or later an issue will arise in your design. Whether this is identified by you, your teammates, your reviewer, or an independent researcher does not matter. Vulnerabilities and limitation are not the end of the world as design can be iterative. Issues might be simple to fix: the addition of round constants to a key schedule; adding authentication to a packet in a protocol; properly sanitising input data for an application. In such cases, admit your flaw(s), make the small change(s) and humbly ask if the issue(s) has/have been fully addressed. In other cases, issues can be much more significant. Some vulnerabilities can be catastrophically exploited and major surgery to your design would be warranted. When such problems arise, it is often best to (painfully) abandon your design and start again from scratch. It is natural that people are reluctant to write-off significant quantities of effort, but some of the worst cryptographic problems have resulted from baroque and complicated designs trying to salvage hope from serious issues.

7. Learn to use some analytical tools

Lastly, it should be clear that improving cryptographic design skills depends heavily on learning from expert cryptanalytic skills. Moreover these skills are a very rare resource which are highly in demand. It can be a while before you convince a professional to devote time and effort to looking at your designs. A less effective but more accessible level of analysis is increasingly available from automated tools. These tools can be difficult to master, but in learning their foibles, you can also gain insight into cryptanalytic techniques. For primitive designs, there are automated tools for analysing $S$-box properties or performing Matsui's algorithm, or for costing the complex array of lattice attacks. For protocol analysis there are formal methods tools such as Tamarin, ProVerif, and CPSA. For implementations there are tools such as Cryptol and SAW. The tool writers are usually very keen to expand their user base and user communities for these will often be willing and able to extend their help in supporting newcomers. The tools may not yet be able to replace the best human cryptanalysts, but their analyses are thorough, "objective" and replicable. Mastering the tools will often teach you a good amount of cryptanalysis.

Post scriptum

Let me conclude by saying that a desire to attain cryptographic design expertise is a laudable aim, but highly ambitious. I consider it significantly harder than cryptanalysis in many ways (in cryptography we need everything to be right; in cryptanalysis we need only a single mistake). To acquire such skill requires significant "social credit" with those able to help you on your journey and the first steps should be to acquire such credit by demonstrating that you understand the power of the sister skill of cryptanalysis. A repeated and annoying assumption from many is that they are a priori entitled to such credit from the cryptologic community.

Answer 2

I'm giving my opinion here as a moderator of the cryptography site where this question was initially posted. I won't provide a TL;DR, but as the question basically asks to join the community but ignore the rules, you may be able to guess what the final tally will be.

How to get your scheme validated is an interesting question and I'll happily upvote answers that go into that direction. This answer will focus on the Cryptography site.

There seems to be a common trend among forums involving the topic of cryptography to shutdown any conversation very quickly, least in my experience.

First of all, Cryptography on StackExchange is not a forum. It is, like most if not all of the other StackExchange sites, a Q&A site. It has a community, but that community should strife for answering concise questions in the best way possible. In a very big sense, it is designed to avoid conversation as that conversation ends up muddling the result for future readers.

From the stand point of a newcomer to the field making bold claims and selling snake oil I can understand the trepidation and warnings.

A newcomer is not defined that way. A newcomer can make basic very basic errors. Of course, the annoyance will be higher if they are making bold claims or sell snake oil, and the response will probably be as loud or bold. But take the student of cryptography that submitted their cipher to the AES competition. That was received relatively well, even if it was broken almost instantly.

But as someone who has been reading the books, studying the courses, and experimenting with code when do they graduate to being able to ask educated questions and get more discourse beyond “thou shall not write your own stuff?”

If a concise and clear question then this should, at most, be a warning in a comment or a back section of an answer. If the question is good then it should be answered. It doesn't matter if the question is simple or outside of what is common. I'm most drawn into discussion here what makes a question good, but for that please read the help section of our Q/A site.

If you find that the question gets closed because "thou shall not write your own stuff" then please flag for moderator attention. This cryptography site is specifically for asking questions that are of academic interest.

Usually though, such a comment would be as a generic warning to the asker and future readers. I hope you agree that starting cryptographers should receive such a warning in advance. That should however in no way stop the question from receiving answers.

For example. I know enough to not write my own cipher. I use battle tested technology like AES-GCM. But what if my experiment wishes to use it in a different way. Maybe a communication protocol for a learning exercise? Or chunking algorithm that uses the cipher for larger payloads?

One thing that this site is not designed for is prolonged discussions about a elaborate scheme for a specific purpose. Doing a full analysis of such a scheme is pretty tough work. It will precisely lead into discussions, adjustments. It is also hard to conclusively answer. Somebody indicates a possible snag, a discussion will start, possibly a new trick is used to overcome the snag, but that doesn't say that the scheme is now secure.

At this point it is important that the Q/A site is designed to deliver answers to anybody searching within the site. This question now took up a lot of time, but it won't ever show up in search results. It is only of value to the person asking the question and maybe a person of two that was following the discourse.

Now you can still come back and ask "so what"? But as a moderator I would then indicate that StackExchange does have a set of rules and as a site we are incensed to operate within the rules. We can, and do challenge the rules sometimes, like we do with homework questions where we accept more than on other sites. However, in my opinion it would be very detrimental to the site if we become "a forum" where anything goes. In that sense SE works differently from newsgroups, reddit and Quora, and I don't think this will change even if site visits are dwindling.

Then again, if you would have this protocol and then ask concise questions about specific parts of it then this is fine. Generally, if your question post contains a single question + maybe one or more related questions then it is fine. If the question count goes to 4 then the question will probably get closed and five is right out. It is important to know how to ask questions on Stack Exchange; just like any other tool, you need to learn how to use it.

When addressing these application level ideas I find it very hard to get any dialogue or information because I’m not allowed to do it myself even if the system I’m targeting lacks something.

If you want to invent something, especially something that goes against established ideas then you need to be very stubborn. It's a trait that you'll find in any field. You do you. Just don't expect this site to accommodate you if you post something against the rules. Of course, being stubborn won't guarantee success - or a secure scheme for that matter - but not having it does probably mean failure.

My deepest apologies if this is off topic to this SE. I’m just at a loss on how to find educational discourse about cryptography and its use without the gate keeping that happens in this community.

As an elected moderator and gate keeper: if you want to do that then do it outside this community. If you want to have a chat or discussion about anything we have the Side Channel, our chat. If you want to make changes within the StackExchange rules then you can create concise, specific proposals here in Cryptography Meta. If you want to change how StackExchange works then there is StackExchange Meta.

Answer 3

In my opinion, the most important factor when doing anything with cryptography is knowing your limits.

Firstly, there's a massive difference between implementing stream encryption using an existing AEAD or implementing Encrypt-then-MAC and designing your own cipher. This is what certain proponents of 'don't roll your own crypto' fail to understand.

If you do sufficient research about a specific problem, you will eventually know enough to design/implement a solution. It's just that the amount of research required and the risk of mistakes scales dramatically as the problem becomes more complicated, like designing an interactive protocol or a new cryptographic algorithm. However, simpler things often aren't as difficult/dangerous as certain people make them out to be.

Secondly, designing/implementing things as a learning exercise (aka not for production) does nobody else any harm. As long as you keep these things offline or have a large warning in the README on GitHub, there's no sound argument against doing this besides that it wastes your time if you're working on something that's clearly out of your depth.

Everybody has to start somewhere, and learning by doing rather than just reading works well whilst also producing something that shows your interest in and knowledge of the subject.

The key to 'rolling your own crypto' is doing it gradually and thoroughly. Don't start by trying to implement AES, implement HMAC. Don't start by trying to implement TLS, implement a file encryption tool or a password manager. Don't start by designing new algorithms, use existing algorithms.

When implementing an existing algorithm:

1. Read about the algorithm (books, the specification, the paper, presentations, blog posts, etc).
2. Implement the algorithm following the specification and/or an existing implementation.
3. Review your code line by line to check for and fix mistakes.
4. Debug the code with breakpoints to check it's operating as expected.
5. Test the algorithm using test vectors.
6. Look at what other implementations are doing differently.
7. Improve your implementation (e.g., the performance, the API, avoiding side-channel attacks, etc).

When designing a protocol:

1. Research existing designs (specifications, papers, blog posts, code, etc).
2. Identify any gaps that can be addressed and research ways they can be addressed.
3. Determine the algorithms you want to use after researching the options and their strengths/weaknesses. Look into their security considerations/how to use them properly.
4. Draft a specification with pseudocode.
5. Review the specification for mistakes (e.g., not authenticating everything) and optimisations. If unsure about anything, do more research or ask someone who's likely to respond (e.g., a blogger who lists their email). Keep any communication concise but with sufficient detail and thank them. You could also experiment with formal verification tools, starting out with something like Verifpal.
6. Implement the protocol using an existing cryptographic library.
7. Review, debug, and test the code.

If you do these things enough alongside wider reading, you'll eventually have sufficient knowledge and skill to call yourself a cryptographic engineer, and people should stop saying 'don't roll your own crypto'. With more investment into the academic side of cryptography, you might become a cryptographer. At that point, you may be able to produce competent new algorithms (secure and with advantages over existing algorithms).

To conclude, I would argue that even before having cryptographic engineer levels of skill/knowledge, one can 'roll their own crypto' (besides creating new algorithms) in production/public projects safely assuming they have put significant time and effort into every stage of the process and are not tackling a problem out of their depth. Ask yourself whether you know enough to proceed and answer honestly.

Answer 4

As a short answer. If you mean that you want to obtain the least amount of knowledge about cryptography to understanding stuffs related to this field, and then tweak them a little bit, I recommend Katz and Lindell's book, "Introduction to modern cryptography" along with additional resources like slides and a free course on Coursera. I believe they wrote this book for almost every audience with every background. The text of the book is easy to grasp.