If this engine and open source will mostly be used to make permissible games, then it is permissible to make it.
If it will mostly be used for haraam things, it is not permissible.
This general principle applies to manufacturing and producing anything that may be used for either halaal or haraam purposes, when it is not known what each user will use it for. Attention should be paid to the way in which it is mostly used by people, because when it is not possible to know about each user, it is sufficient to know what is usually the case. If what is usually the case is that it is used for haraam purposes, it is not permissible to help with it, because of the prohibition on helping others in sin and transgression.
It says in Fataawa al-Lajnah ad-Daa’imah (13/109): Everything that will be used in a haraam manner or it is thought most likely that it will be used in that manner, it is haraam to manufacture it or import it or sell it or distribute it among the Muslims. End quote.
And Allah knows best.