AEM Custom dialogs facilitate the work of your content authors by extending the classic AEM dialogs with custom functions that are not possible with AEM. A very good example is the icon picker: The content authors have a selection of icons which are available via a dropdown. To get a visual feedback, the authors first have to select the icons, save the changes and then open the dialog again if the icon does not look the way you want it to. Custom dialogs can be used to open another window by clicking on the dropdown, in which the icons can be displayed and selected by clicking.
Another use case for AEM custom dialogs occurs when the complexity of a component can only be represented by the AEM dialogs with difficulty or in a confusing way. An example is the creation of a REACT table, which can have any number of columns and rows and each cell is to reflect different functions (image, icon, tooltip, text, and so on). The mapping with classical AEM dialogs is possible with the help of nested multifields, but this kind of realization is not only extremely confusing and error-prone due to the complexity, but also reduces the user experience of the content authors into the bottomless. Custom dialogs can be used to create a REACT component for maintaining the product table, which can reflect the corresponding complexity with a very high user experience.