As mentioned in the comments, you need to prevent form submission when the dialog box is shown because the dialog box does not block the UI. It will continue to submit unless you stop it. After you press the button in the dialog box, you can actually submit the form.
Now the tricky part is that when you actually submit the form, this also triggers the onsubmit
function again! A good way is to set a flag. See the pseudocode below which should basically do what you want.
<form id="orderForm" action="/mywebsite/order.htm" method="POST" onsubmit="return (validateOrderForm(this) && validateDialogForm(this))"> ...
let real_form_submit = false; function validateDialogForm(theForm){ if(!real_form_submit) { $('#checklist_dialog').dialog({ ... buttons: { "SAVE": function() { $(this).dialog('close'); real_form_submit = true; theForm.submit() }, "CANCEL": function() { $(this).dialog('close'); } } }); } return real_form_submit; }