Neutron diffraction is one of the three crystallographic techniques (X-ray, neutron and electron diffraction) used to determine the atomic structures of molecules. Its particular strengths derive from the fact that H (and D) atoms are strong neutron scatterers, meaning that their positions, and thus protonation states, can be derived from crystallographic maps. However, because of technical limitations and experimental obstacles, the quality of neutron diffraction data is typically much poorer (completeness, resolution and signal to noise) than that of X-ray diffraction data for the same sample. Further, refinement is more complex as it usually requires additional parameters to describe the H (and D) atoms. The increase in the number of parameters may be mitigated by using the `riding hydrogen' refinement strategy, in which the positions of H atoms without a rotational degree of freedom are inferred from their neighboring heavy atoms. However, this does not address the issues related to poor data quality. Therefore, neutron structure determination often relies on the presence of an X-ray data set for joint X-ray and neutron (XN) refinement. In this approach, the X-ray data serve to compensate for the deficiencies of the neutron diffraction data by refining one model simultaneously against the X-ray and neutron data sets. To be applicable, it is assumed that both data sets are highly isomorphous, and preferably collected from the same crystals and at the same temperature. However, the approach has a number of limitations that are discussed in this work by comparing four separately re-refined neutron models. To address the limitations, a new method for joint XN refinement is introduced that optimizes two different models against the different data sets. This approach is tested using neutron models and data deposited in the Protein Data Bank. The efficacy of refining models with H atoms as riding or as individual atoms is also investigated.